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ABSTRACT ■ - . 

The first paper in this collection of symposium 
papers presents the results of a survey of nearly ,600 educatioilal 
institntions 'and provides information on institutional', 
chSiracteristics^ institutions engaged in teacher education 
evaluation,- scope" of data collected, methods of, data collection, 

* relationship between size of i^ogram and method* of ^ta collection, 
perceived value of teacher*^education eifll^luation^ perceived needs in' 

* condnctinc^ teacher education evaluation^ and^repocted budget for ' 
evaluation* Tn the second paper, a discussion is given on the broad 

, spectrum of evaluation and ^'don'ts" for .program evaluation* The third 
P^per of f ersdteuqgesti^ns "^or . planning* developing^ impleiaenting, and 
* operating*an evaluation program* Ten design characteristics irhich^ 
sho!ild be considered for developing an evaluation program are 
described i^i the fourth paper* The fifth paper presents ways to^ 
modify costs of program evaluation and'suggestions for taking 
advantage of existing resources* The concluding paper addresses the 
A future oi teacher educaition evaluation* Brief summaries report on 
interactions in small discussion groups at the symposium at which 
these papers Mere presented* The concluding portion of the monograph 
, ^is an summary from a repi;esen<^ativ^ of the National Institute of 
' education* (JD) ' ' * 
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FOREWORD 

The R&O Center's mission is to design and conduct research and related 
activities in" the fie^ld of teacher edycatidn. This 15 carried out in close 
collaboration with practitioners, policy makers and other researchers who are 
most cognisant Qf emergent unsol ved problems and chal lenges in the field , 
/Through its own research, the Center is contributing to the knowledge bases 
- underlying 'effective te^chifig and learning in various college, school and 
classroom contexts, effective teacher education systems throughout the career 
cycle, 3nd successful implementation of re search -based organizational and 
instructionaV )jr act ices, ^ 

In keeping with its role as one of the national centers for research and 
development in education, the Center attempts to f aci 1 1 tate and coordinate 
various p>awnjng^ networking and collaborative activities which link commijm- 
tjies of scholars and practitioners both nationally and internatiooal ly- One 
interesting e*(ample of an informal, and spontaneous network which ^as develop- 
ed over the past few years has involved individuals from across the country 
working in relative isolation on the design and conduct of teacher education 
program evaluation fol low-up studies , This network (TEPFU) was of f ici al ly 
"born" in April, 1978, when the Center hosted a national conference ^f indi- 
viduals who| had conducted such programs or studies. Conference proceedings 
were pLjblishjed, and the Center began to receive numerous 'inquiries for fur- 
ther information and requests to be aff 1 1 iated wi th further meetings and 
othem information sharing opportunities. A newsletter , was developed ^s one 
5ay of responding to this increasing interest. Over the past three years, 
meetings of the expanding group have been held in conjunction with annual 
meetings of appropriate scholarly and professional associations, and these 
have provided research reports\ professional dialogue and increased member- 
ship in the network, ^ 

We believe th«t increased attention to systematic evaluation of teacher 
preparation programs irtcTuding appraisal and analysis of graduates' per for* 
mance arid experience tht^ough their first professional ^ar is an invaluable 
source of feedback id the^redesigning, refinement and enrichment of prepara- 
tion programs. Me are rioting increased sophistication in the'design of such 
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studies and in solving the many logistical and economic problems which attend 
the implementation effort . The R&O Center i s pleased to play its part in 
f aci 1 i tat ing comrnumcat ion and some emerging synergy in^ the communi ty of 
scholars engaged m this important work, 

0] iver H, Sown 

Director . . ■ 

Research and Development Center for Teacher Education 

The 'Jnwersity of Texas ^at Austin 
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INTRODUCTION ' 

Ah increasingly import ant '^problem ^faced by schools, colleges and d^e- 
partments o'f e'ducation, is evaWation of' their teacher preparation programs* 
Interf\al pressures encdurage institutions to demonstrate program effective- 
ness; external mandates for meeting accreditation standards reinforce the 
need to engage in evaluation, ^WithOut' well dev&lopecf program assessment fol- 
>owed by systematic implementation of the^changes warranted by the examina- 
tion, ot IS difficult* to imagine how program eff;ectiveness may be iiDproved\ 

Over the last decade there has been a slow but growing interest and in.- 
volvementin teacher education program evaluation. The Jandefur (1970) model 
provided a cornerstone for the development of evaluation studies early in the 
197D' s by insti tut ions sucH as Western Kentucky Uni versity (1972) and 
Tennes see Technological Uni vers ity (1974) . In several other areas of the 
natiorr and- in Canada (Dravland & associates), others were also beginni?ig to 
develop strategies for assessing teacher education programs. The National 
Counci 1 for Accreditation of TeStcher Education (NCATE) . standards further 
encouraged--insisted on--prografn examination. * ^ , 

Then in 1978 wre focused attention occurred through various organized, 
structures. The annual meeting of the American Association of Colleges for 
Teacher ^Education (AACTE) that year devoted one of* eight "^program "strands" 
which.xut across the four day meeting to follow-up studies of teacher prep- 
aration programs* In the spring of that year, the Texas Research and 
Development Center for Teacher Education became involved in program evalua- 
tion efforts in a response to the needs of individuals and institutions en- 
, gaged in, evaluation and foUcJw-up studies* A collegium of eight institutions 
met under the, auspices of the Texas R&D Center in order to^ share techniques, 
concerns, problems, and successes. This small but very successful attempt at 
1 inking ^persQris involved in program evaluation provided the forum for instj- ^ 
.tutior^s'to share what^ they were doing. Several reSuUs of this working con- ^ 
ference contributed to ttje increased viability of program evaluation* 

A monograph containing the reports of the eight institutions' evalua- 
tion ^todies was published (Hord & Hall, 1979)* This* volume has had wide 
dissemination from the R&D Center and provides a practical jreference for how- 
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to-do ' such studies, -as reported by those who have had 'extensive experience, 
A second importani outcome of the conference was the initiatrion of a network 
for persons who wish^ to communicate and continue to share with^each other,. 
This network is known as TEPFU,^ Teacher Education Program Follow-Up. A very 
i nformal newsletter, publ ished three times^ a year by the R&O Center, was 
institvj'ted, and serves as a means for /sharing inforjnation and disseminating 
products to the TEPFU network members , the membership of this netwovk 
continues to expand. . * * 

Following these activities in 1978, there was an interest in, but a lack 
of funds for supporting ways that woul,d permit individuals to engage in per- 
sonal interaction and dialogue. White the newsletter kept people connected, 
there was lacking a vehicle for making discussion possible. Consequently, 
small 3d hoc gatherings of program evaluators *{and those interested m so 
becofning) were held m conjunction wUh various professional organization 
meet^ings^- 



In ]960, external pressure on ins tiT%/t ions to do program evaluation was 
escalating. At the annual AACTE meeting that year the small space accorded a 
session on follow-up evaluation of graduates, anchored by Adams and Ayers, 
could not accommodate the large number of peopl^ trying to attend- Persons 
who could get in indicated interest in staying' in touch and participating in' 
some inexpensive^^emi-for^mal strategies for the purpose of sharing with each^ 
other and acquiring information related * to program evaluation. Several 
months ^^ter at the annual American Educational Research Association (AERA) 
meeting, a group of persons agairf met informally to generate such|i<jeas; some 
very promising strategies for making program evalU-ation efforts more visible 
were identified and some activities to respond to the needs and interests of 
persorys involved ^in or expecting to be invoTv^^l in program evaluation were 
suggested. These events reflected a substantial vigor in this area of in- 
quiry. 

Because of keen^ interest and responses from the teacher education pro-* 
gram eval"uation network that reflecte^d continuing need/ for assistance, a ses- 
sion was designed to be a part of the 1981 AACTE annual meeting. As a result ' 
of the 1980 experience a better case was made for more space and time to be 
available for TEPFU activities. Thus, a, program format that included a large 
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block of time was arranged in order to address issues (e,g,» accreditation), 
share experience and expertise* and engage participants in meaningful activi- 
ties targeted at improving their own skills %and capabilities for undertaking 
their program assessment. The AACTE staff and program committee facilitated 
the arrangement of program time- and fleeting space for such a session,. 

The session was designed in two parts: (1) six invited papery focusing 
on issues of program evaluation were presented by the authors; (2) 4II ses- 
sion attendees were organi zed into smal 1 di scussion groups fbr interaction 
and di a1ogue» stimulated ^ by the paper presentations . - Members " of the TEPFU 
network guided small group discussions. As an outcome of each small discus- 
sion group, a statement of response reflecting their insights* reactions, and 
questions was to be prepared. These statements collaboratively produced by 
participants of both larg^ and smal 1 insti tut ions » by persons 
with some years of experience and others with none* would repi^esent current 
experience and thinking "in the field." 

The pyMication of this monograph* Teacher Education Program Evaluation, 
1981: Theorj^ and Practice , further testifies to the ^mportai^ce of that ses- 
sion. It inclXides the six formally prepared papers and the summary papers 
which resulted from the work of the small interactive groups^ This smal 1 
volume will acquaint the reader with current key issues in teacher education 
program evaluation as well as provide assistance for those engaged in plan- 
ning, implementing or operating evaluation programs. 
Part I 

Part I of this monograph is the set of papers prepared for presentation 
at the session. The, first paper by Adams and^Craig reports on current prac- 
tice in evaluation efforts. A survey was (Jeveloped\ at Western Kentucky 
University and sent to nearly 800 institutions in order to reveal the' state 
of the practice in program evaluation. Responses to £he^ suVvey are reported 
by the authors in eight categories; inslfitution demographics, institutions 
engaged in teacher education evaluation, area of data collected, methods of 
data collection, relationship between size of program and methpd of dat^ col- 
lection, perceived value of teacher education evaTuation, perceived needs in 
conducting teacher education evaluation and reported budget for evaluation. 

In the second paper, Lyn Subser suggests "a dozen don'ts" for what not 




to do in program evaluation. From his perspective as director of the 
National Council for Accreditation of Teacher Education, Gubser discusses the 
tiroad spectrum of evaluation and summarizes with how-not-to-do-its for peop^^e 
^interested in establishing successful program evaluation, 
J In the third paper, Craig and Adams suggest a process for pi anniog, 
'developing, implementing, and'^'Operating an evaluation program that will ""^ffiax- 
' imize the utilization^' of evaluation data in program decision making, 

A critical aspect of the design suggested by the Craig and Adams pap^T^ 
is more specifically addressed in the fourth paper by ^Oerry Ayers, Tennessee 
Technological University, Ten design characteristics which should ,b€ con- 
sidered for developing an evaluation pr65ram are described, , as well as prob- 
lems which may be enct)untered if the characteristics are left unregarded, 

Ede 1 1 Hearn reports on Tennessee Technological University's evaluation 
J^ystem and its cosVs, Effective evaluation *of programs should allow appro- 
^priate modifications in programs, but this, can happen only if finance and re- 
source" allocations -for the evaTuation are part of'the institution's budget. 
Ways to modify costs and suggestions for taking advantage; of existing 
resources are presented, " . ^ ' 

The concluding paper addresses the fut^ure of teacher education evalua- 
tion. Gene Hall, Research and Development Center for Teacher Education, 
raises questions about the future of teacher Education program evaluation and 
foUow-up studies as ,a whole, * He distinguishes between issues that are. con- 
trollable and those that are not, ^ He makes concrete recomrflendations for the 
future regaining the control I able areas , and speculates on the pressures 
which may be anticipated in the areas not under control, ' 
Pa rt II ' ' 

Included in this section of the monograph are the brief summary reports . 
of^the interaction by the participants in the small discussion groups*. Their 
reactions to issues raised •by the paper presenters were synthesized by the . 
group facilitator* More than 200 persons attended the session^artd 7Ji ne*.^ 
^mained into the'^dinner hour ta^partiCipate in the group activity. 
Part III ' ' ' . - !" / 

The concluding portion of this mono^g^raph is an executive sunJnary by ^oe 
Vaughan of the National Institute of Education* Vaughan provides ^a synthesis 
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of .tfi ernes of the <)apers and group statements to establish a composite image 
of the issues', qirestions and concerns which "emerge-from al) the writings. 

.It is the hope, of^^^ who were "involved in contributing to this^mono- 
(graph .that it will prove t« be" instrifCtive, providing useful information to 
program evaluation practitioner^. Each of thf^ authors Invites the readers to 
^contact them for further did^gue and interaction. Despite the difficulties 
of doing program ^evaluation/ it^s apparent that for the present, teacher ed- 
ucation-program evaluation lives. " . 

Shirley M". Hord 

Reseai^ch and Development Center for Teacher Education 

Ronald D. Adams ' * . " . ^ 

Western Kentucky University , 

Hay, 1981, 



A Sl^VEY OF UNDERGRADUATE ' ' \ 

TEACHER EDUCATION EVALUATIOI< PRACTICES 



(Ronald D. Adams ' , 

'James R< *raig 
Western Kentucky JJalversity 

# * 
^ . - _ ' ' ' K INTRODUCTION 

Several factors have led to our conduct of a survey of teacher edu<;a- 
tfon evaluation practice. 'First, due to our' experience irf teacher ^edyf^tion 
evalu^t-4on since the ^tW of 1971, we have had the oppi>rtuni,ty for dialogue 
with numei'ous ind^\ficluals who are planning' or are engaged in evaluation pro- 
gr^ams. Many of these individuals have expressed a desire* to know the state- 
of-the-T*ractice m teaqher educSition evaluation. 

Second, the growth^of the TEPFU (Teacher Education , Program Follow-Up) 
group and the high attendance at national ''meetings where tgacher education 
evaluation topics were presented has demoqstra^tTed a continuing interest in 
this topic. One of the reconwiendat ions from the TEPFU group that a study 
be conducted to determine ^e current^ evaluation^ practtte in teacher edqca- 
tion evaluation as there appears to be little research in the professionaT 
lit^ture on this topic, 

\ Third, the NCATE stahdar<Js fot* accreditation have explicitly stated that 
institutions producing teachers (nust have a systematic '^aluation of their 
teacher; education programs to^ include gathering data while students ^re ,in 
the preservice program^and 'after they'have entered the teaching profession' 
Evidence must also be prov.ided of utiUzatfon* of data for program improve-- 
ment. JJndoubtedly, these standards ha^(e provided *much of tjie interest in 
teacher education eyaluation* - - ^ 

Finally,>^in a 1978 doctoral dissertation ^t the University of Tulsa, 

Wennette Peguqt reported a survey of , teacher education follow-ufi evaluation 

.... > . , ' ' ' 

practice. This study providecf a. beginning for determining the status of 

teacher education evaluation practice. Two of her- findings have particular 

iliterest to this study* First, she concluded that, "There is only moderate 
^ , s ^ 
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comtemporary involvement in systematic fol low-uP evaluative practices by 

teacher edijcation programs." Only half of the respondents .reported that they 

were engaged in this practice. ' Second, she.' concluded that, "There is a 

fninimal leveT Qf evaluative sophistication in the design and the 

iriiplementation of comtemporary evaluative schema," The most prevalent schema 
' ' =^ > ' ' 

was the cpntroversial one-shot questionnaire sent to graduates the fir'St year 
fallowing graduation, * * ^ 

From , the pr.eceding dis^cussion, it seemed clear that a current surviey of 
t^eacher education evaluation practice could prtivide Tneaningful information 'in 
identifying needs Vnd future di/eciions in t^>is area. An initial survey form 
was developed by the authors to obtain information from the following areas:^ 

1 } Institutional demogt^aphics 

2) Topics on areas evaluated ' \ 

3) Techniques used in evaluation system ' * 

4) Sources of revaluation data 

5) Points of evaluation 

6) ^1^^ relative to e\raluatitin practi,ce 

7} ValOe placed ort evaluation ^ ( 

8^ Resources devoted to evaluation \^ * ^ ' 

The Initial survey form was reviewed by college ^inistrators at WKH and a 
later revised version was sent to selected members of, the TEPFU group^ via 
assistance frorrl the R&O Center for Teacher Education, Finally, the third 
version was sent to AACTEWjd NCATE officials for input. 

All suggestions were considered', final changes were made, and the survej^ 
forms were^mailed about the middle of November, 1980. Even with a short re- 
turn time, the holiday season/and the rather lengthy form, a 36 percent re- 
turn rate was obtained by the end of' December, A follow-up survey was sent 
about the second week in January, 1981, resulting in an additional 20 percent 
return for a total of 442 returned survey forms to date> 

To prepare for this report, data obtained from 397 forms (51 percent) 
were processed. The rema'ining wi 1 1 be reanalyzed for a^ later publ icat ion . 
Thi^ more complete report- will contain detailed analyses, and findings aOd 
should be available during the summer of 198K \ ^ 

Time constraints do not permit presenting in great detai 1 the results 

, 10 ^ ' . 
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of this survey--in ^act , much of the data have yet to be Analyzed beyOnd the 
rudimentary findings prepared for this paper, However, the findings present- 
ed here should provide a bas is for understanding more fuLl ]y the present 
state-of-the-practice^n teacher education evaluation, an^s^ggest direct|oh;s 
we must take as a- profession if we are to improve ^feis ,e^s^eWtiaV 



praotice. 
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Th? f )ndings presented for this report' represent preliminary analyses 
.of the data obtained from 397 responding institutions, Thjrty-eight addi- 
tional questionnaires have been received s^ncp these analyses wer^ conducted 
and are being prepared for inclusion in the data set. Nine institutions 
^did^ hot qualify for inclusion for analysis due to such factoids as having no 



undergraduate programs or no longer being in operat ion . 
responses totaled 444 for a return rate of 57, D percent. 
The findings are presented in eight categories: 
A, Institution Demographics . ^ 

Institutions- Engaged in Teacher Education Evaluation 
Area of Data ColTected 



The accounted for 



D. 



Methods of^ta Collection 

Relationship Between Siz^ of Program apd ilethod of Data^ 
Collection 



F* Perceived Value of Teacher Education Evaluation^^\^ ^ ^ 
G, Perceived Needs in Conducting Teacher Education Ev^lud^ion 
Reported Budget for Evaluation 
Undoubtedly, additional analyses will provide more information regarding the 
^ate-of-the^practice in teacher education evaluation than these preliminary 
analyses. However, we feel findings from these analyses provide more insight 
into what is being don§ in teacher education evaluation than ha^ been known 
to date. 



^ Due to". space limitattons the tabular i)resentation oP data will be 

'limited ii> this refjort. ' - 
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A, Instityt'iort OemogT"aphks ^ - ^ ' 

The responding insfUutions were evenly split between public and private 
with 52.4 percerit public and 46,3 percent^pri^ate, NCATE accredited institu- 
tions comprised 63.7 percent of the sample while 82.6 percent reported mem* 
bership in AACTE,^ Geographic region represented by responding institut.ions 
was indicated by membership in regional a:cci^edi t ijog agencies and can be seen 
Nin Table K*- Institutions . in the. North Central -Associatioo comprised 48,4 
percerit of the ' respond'e'hts,. followeiJ by the. Southern Association with 27.4 
percent, and the Middle States Association wit+i 11.5 percent/ These data are 
v^ery'close to the percentages obtained by Pegues in 1978, 

. Institutional size ranged f^om 300_to 63,115, with undergraduate teacher 
' edacdtyTn programs ranging from 13 ta 7,300 students/ These dat^a' are pre- 
sented in Table 2. It can be observed that 57.7 percent of the programs have 
500 or less enrollment. Irrs^utions produrs^ing only 8achelOr*s Degre^es made 
^p 31.8 percent^ of the sample whi)e 2'2.0 percent of the responding institu- 
tions offer the Ooclorate. ' ' * . 
6. Institutions Engaged in Teacher Education Evaluation 

, Two categories of data, size^of u/idergraduat^e teachfer educatibn program 
djhi publ ic/pff i vate. cllssif icatioft were cross tabulated with institutions * 
response to Question 1, Part^ H of the survey form, Thi s^ question asked 
^institutions to indicate 'if the> were engaged 4n evaluation of their under- 
graduate teacher education V^i^f^'^^ ■ ^^^^ ' responses it was determined 
tjiat 86.7 percent of*"iH=ve—iflit i tiJt ions indicated some evaluation activity. ^ 

cross tabulation of size pf program and public or private classifica- 
tion indicated a relatively strong Relationship between these two \?ariables. 
Smaller programs tend to te in pr^ivate institutions while larger programs 
tend to be i/i publ ic ins t1 tut ions . Whi le this relationship is not sur- 
prising, it should be kept in m'ind whef> discussing other analyses* 

Three categories of size were utilized in the cross tabulation analysis 
with those reporting evaluation" pract ices . . They were a^s follows; 

small - ^ * "^^^ students . * 

medium i 251 to l.OOO^student^ . ■ ^ 
large -^1,001 to 7,500 students " ' 



Figures appear at the efidr of each paper. 

, ' * " "l2 "" 
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There^appegred to be little difference, J/) the^percentage of institutions re- 
porting evaluation- practjice and the size of the teacher education program. 

Similar'ly- wh^n public or private c^lasslfication was cross tabulated 
with reported evaliiatibrf practice no meaningful differences were. noted." This^ 
was expected due to the relationship betwp^n fJS'Cgram size and public/private 
classif IcatioW: ' ' " ' ^ ^ 

C. Area of DHa Collection . / - * 

Institutions were 3Sked to respond to a JUt of Seven evaluation areas 
for which they cbllected d^a. -they were instructed ^to check each area for 
which they collected data. The "Teaching Ski J 1 s,' Strategies, and Techniques" 
received the most checks, with ''Knowledge of Subject Matter" ar^ "Relation- " 
'ships with St'^dents'* ,fol lowing,' respectively. The least checWti items were 
"Relaticioship with Parents" and '^Participation in Professional Organize 
t^ons", respectively. 

Additional analyses are plann^(* to determine if there relationships 
between evaluation area's and other "var i ables . For example, the Question "are 
method and ^rea related?'**may provide some insight into source of data col- 
leci^n and what 15 being collected. 

D. Hethcids of Data "Collection - * . ' 

The major focus of the survey^was to determine the state-of -the-practice 
in teachar education evaluation. A major component was the methods efnployed 
to follect data to include type of data collecting devices or procedures, 
source of data and frequency or points of data collection. Institutions were 
asked to respond to seven data collection procedures* eath having ^matrix-of 
data source by col lection frequency . Through this reporting procedure it 
cOuld be deterfoVned the most often utilized source and the- frequency with 
which data were ^col lected from sources for any given proceduna^^^ 

The seven data collection procedures contained in the survey were: 

1 . Quesiionnaires 

2. Personal Interviews 

" 3. t)irect Classr^m Observation ^ , 

4. Media-Aided Work Saijiple Observation . 

5. Standardized Tests 

'6. ProfessiOjial Competence Measures 
7. Personal Charaicteristlcs and Attitudes Measures 

13 ^ 
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Resppndents were asked to place tfie approxiitiate percent of |>rogram enrol Iment 
sample for each data collection period and eacfi source, yhile the majority 
of respondents followed Jthese directions, several returned survejfs had only 
checks* in the matrix grids. Thus, for these preliminary'^'analyses, ..sample 
size was -not considered; rather, the frequency of participation only was 
establ ished, 

4 

Data, from these questions provided '*maps'* of evaluation practice fhat 
describe "the methods utilized in teacher education evaluation. While these 
analyses provided only .summary data on "methods of evalu,ation, they allow for 
some generalizations to be drawn. 

■ Questionnaires were by far the most utilized fo^^ of collecting evalua- 
tion data-, both in preservke af^ fol low-up. The source most often given'^^as 
the student/graduate and the most frequent* pOfnt of data collection was at 
ex^^P^rom program {65,8 percent). The Supervisor was the second jnost used 
source for evaluation ^data with peers providing some data in the preservice 
pi^ogram, Tabl% 3 presents these data. ^ . * * — ■ ' * 

When a usage criterion of "10 percent was- utilized to 'examine the re- 
sults, It was determined that data wer^ -collected from student/graduate^and 
supervisor at entry and continued .through the, 4-6 year of follow-up. Peer 
data were only used in the preservice program ^in categories "during" and 
"exit", The heavy black line indicates the ceHs which met the usage cri- 
terion, ^ 

Personal interviews were used to a much" 'lesser extent than question- 
naires, and were limited primarily to yse in the ' preservice program. The 
student/graduate and supervisor were agaio the -primary sources. Table 4 con- 
tains the summary data on personal interview.' ^ 

Use of direct classroom observation as a technique for collecting evalu- 
ation data was quite limited except in preservice programs. Students/ gradu- 
ates were observed in many programs during and at the end of teacher jeduca- 
tion programs, and to a very ,lim,i ted' extent inuring the first year of .teach- 
ing* Pupils were observed mostly during the undergraduate preparation pro- 
gram. These data are* presented in Table.5. - 

The method using ^rj)e3Ta-aided work sample observations produced a 10 per- 



cent usage in only the '^during" category of the preservice program* Table 6\ 
shows these data. ' _ - ' ^ ^ ' 

Use of standardized tests were also^quite limited and were restricted to 
□ seJn the preservice program. This type of data was primarily collected at 
entry" and at exit to the' undergraduate jieacher education program, as can be 
seen in Table 7. - ^ * 

The last twQ categories of. data collection provided quite similar re- 
sults. Professional competence and personal characteristics and attitudes 
were probably recorded as extensions of questionnaires by many respondents. 
They reflect usage during the preservice 'and first year follow-up with lim- 
ited usage the third and fojurth -sixth years of follow-up. .PMmary sources 
were the student/graduate and supervisor, LimSted usage of peers as a source 
was noted during preservice. Tab4e^ 8 and 9 contain the summary data "for 
t'hese van ables . 

Ip summary, al 1 techniques were used to varying degrees in preservice 
evaluations. The sources, however, wer§ 1 united to the student /graduate, 
supervisor, and to a lesser degree peers, FoUow-up evaluation, was^ conducted 
less often than preservice evaluation. . Also, fewer techniques were widely 
used, for follow-up evaluation with questionnaires being by far the most often 
reported technique. As with preservice, the sources were mainly limited to 
student/graduate and supervisor, 

E. Relationship Between Size of Program and Method of Data Collection 

Siven the extreme range in size of teacher education programs reported* 
the authors decided to investigate" the relationship between size of prograrji 
and methods of Evaluation reported. Chi ysquare analyses wer^ -utiTized to 
determine if a relationship existed betwee^ the two sets of variables." 

' Firsts cross t-^bulations were made between size (broken into three cate- 
goriesj and overall indication of use for a given method or technique. Chi 
squares were computed for each of the seven techniciues and size. Qnly two of 
, these analyses approached .significance at the- ,05 level of confidence.. It 
was observed^ that direct classroom observation and media-aide^d work sample 
observations obtained a large enough chi-square value to approach or exceed 
the significance leve^l of ,05 when compared to program size. It would, appear 
^'hat in both cases, ,the smaller programs reported more use of these tech- 



niques than the l^dr^geV ^ programs, 

A second set :-of-'^^inalyses dealt only with follow-up data.^ Due to the 

^sfnall fregUency jDf use» only questionnaires; -personal interviews.* and direct 

■g^assroom observa^tions wer^incTuded in these analyses. However, the analy- 
sis covered the first, Ihvrd and fourthv^ixfcfr years of follow-up for eac'h 

. technique . ' ' ^ \ 

From these' analyses an interesting find-ihg ^emerged. There appeared 16 
be a pattern <Jf usage fo^^ techniques and ^prograoiVsize . Four of the fiveichi 
square ^narlj^es were "significant beyond the ,05 f.evel^of confidence for 'the 
flrst'yea^' follow-up data. There appeared to'be more usage ^among the larger 
programs^of questionnaire, personal interview, -and^direct classroom observa- 
tion- than among the Sfrial ler programs, Tl^i s ^ wai !an opppsi te Mndicati6n as 
f ound ' for direct classroom observation and medi'a-jaided work sample observa- 
tion for total evaluation usage, , ^ 

It al so^ appeared that by fetne third year and beyonds the u-sage of these 
techniques cjiminished to tt^e , point of no differences between programs of 
varying size, i,e., for each si2<?'of program ilassification* ther^ was little 
bei'ng done for follow-up evaluation, ' * * 

F, Perceived Value of Teacher Education EvaJuatioo 

' AACTE institutional representatives or their designees were asked .to 
complete the-survey formv It was noted tKat for the most part administrators 
completed the questionnaire^ In the "otFrer" category, the most often given 
titles of the respondents were d}rector pf student teaching director of 
the research unit, 'Thus, the perceptions of the value of ^teacher education 
evaluation" was primarily 'from an administrative viewpoint, ^ 

/ Per^^'ptionSr'lSr the value of teacher education evaluation were- compared 
f or' Xbos^ ' insti t|uti_ons who had- evaluation and those who reported no evalua- 
titin activities/ While oo formal statistical analysis w^s computed to deter- 
nrine . the signjil'icance of the difference, it appeared t+iat the institutions 

hav>/ig evalualtion programs valued teacher education evaluation more than did 

■ i * 

the non^eval Ration institutions, However , both ' gro^jps tended to perceive 
evaluation a^' fnore positive than negative. 

*Additiortal anailyses were conducted to determiine if size^ of program^ was 
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related to perceived value of evaluation, Chi squar'e\a5p^J.ysi$ wjs utilized 
with both the institutions reporting evaluation progVams; and thnse not re- 
porting evaluation programs, ^ ^,r.\^ * 

In both cases there appeared to be ho relat ipT^ship found for administra- 
tors' perception of value .of evaluation artd site %f program. However^ in 
both groups there was a significant chi square found ^for perceived value of 
evaluatipn by facult^ and program size. Faculty ir> the smaller programs were 
rated as valuing evaluation rnore than faculty in the larger institutions for 
both groups. Students were also rated as valuing -evaluation more for thos^e 
institutions which reported having evaluation programs. It should be kept in- 
mindj how^ver^ that the raters for the most part^ were administrators. 

G. Percewed Needs m Conc^uctlng Teacher Education Evaluation 

, Respondents were also asked for perceived needs in conducting evaluation 

of teacher education prp^ams* Agam^ those responding as having evaluation 

.programs and not having evaluation programs were informal ly compared . 

*' Developing Instrumentation'* was*^ a frequently given nee'd for botrh groups, 

ranked 3rd for the evaluation group and 1st for the non-evaluation group. 

Assistance with '^Planning and Implementing a Feedback System''^ and *'Uti- 

1 1 zing Data for Program Improvement" appeared to be the greatest need for 

inst 1 tut 1 ons having evaluation programs > wh i le *'P1 anning an Evaluation Sys- 

- ■ ' » ' * 

tem" and ''Identifying Instrumentat ion" .ranked number two for fnstitut-ions not 
having^an evaluation system, 

H, Reported Budget far Evaluation ' ^ ' \ ^ ' 

To estimate the resources being allocated to teacher education evalua- 
tion^ i/i$titutions were^ asked to give the approximate^ operating budget for 
evaluation of teacher education programs, Th^ range in budget aHocatiOns 
was from $15 tO-$8U610,^ Slightly more .than 50 percent of the institution^ 
reported spending less than $1^250 on evaluation* However > 18,5 p^ercent re- 
ported '^spending over $10»000, The mean ^nd median for all institutions were 
S6, 280*20 and $1^202.50, respectively." 

Institutions were also asked to indicate the source of funding for/thei> 
evaluation program* Of those responding, 3.4 percent or 32 institutions 
indicated some or total external funding from public sources. In addotion^ 
K2 percent or four institutions reported funding from private funds. Most 

" ,7 ' ' . i 



of the ^nsti tat ions'* that— rep6r ted , funding" -source (n=239) , imjkated total 
internal funding" for their evaluation programs" (92^,9 percent). Those not 
reporting funding source totaled 100, . Of those institutions reporting 
.funding sodrce$, 23.3 percent indicated funding on a one^ year bas.is, whi)e 
71 percent .indicated funding was- recurrent . 

HI, SUMMARY FINOINSS 



The following conclusions were made fr*om' the preceding preliminary 
analysis: _ ■ ^ , . , 

'1, There ^ppearecT to be little difference in the percentage of institu- 
tions reporting evaluation practice and the -size of enrollment in - 
the -teacher- education program or whether the Institution was' public 
or private, ^ 

2, Th$ areas of. "Teaching Skills, Strategies, and Jechmques" , "Knowl- 
?dge of Subject Matter" and "Relatioaships with S'tudents" were the 
most frequently evaluated, ' . 

Questionnaires were by far the most used method of collecting data 
with the studeht/gr aduate the most frequent reported source of 
. ■ datav \; ' , ' 

4, More evaluation practice anji more varied methods of evaluation were 
reported for preservice evaluation than^'for follow-up evaluation. 

♦ 

5. Smal ler institution! tended to use Direct Classroofrr Observations and 
^ ' Media Aided Work Sample Observations more than larger^nstitutions.^ 

However:",' this technique was most offcen reported in preservite evatu,- 
ation practice. ' " , 

6*" When only fo) low-up evalu^ticm was considered, there were differ- 
ences, noted thft firs.t j^ear, but not the third or fourth-sixth years 
for Questionnaires, personal Interview, and Direct Classrodm Obser- 
vation*,, The larger institutions appeared to be engaged more often 
than the smaller instituti>>ns for the first year follow-up, but no 
differences emerged after the first year** There wasya tendency for 
.smaller institutions to do more follow-up at the Tourth-sixth year, 
point. ' : , 

7, From the college administrator perspective, the following values 
were observed: 

■ a. Institutions who responde^J that they have evaluation programs. 
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valued evaluation practice more highly than institutions who 
, reported no evaluation programs, 

b, Size of program was not related to the degree admi/iistrators 
valued evaluation practice, " t . 

C: Faculty and students in smaller institutions were perceived to 
value evaluation practice higher than their counterparts in 
larger institutions. 

Programs that reported evaluation practice most of ten ^perceived the 
following n^eds, respectively: 

a. p'lanning and Implementing a Feedback System, 

b. Utilizing Data for Program Improvement. ' 

c. Developing Instrumentation. " * * 

Programs that rfid not^report an evaluation program perceived the 
following needs most frequently: 

a, Developing Instrumentation. , / . 

b, Identifying Instrumentation, ^ . . " ^ 

c, Planning ^an Evaluation System, 

Programs reporting operation budgets varied greatly in the amobnts^ 
of suppt^rt. In nearly all cases* funding was from internal funds and 
most reported recurring funding. 
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' ' TABLE 1 
Summary of Institutions by Selected Demographic Categories 

Category ^ _f_ - _% 

. Public ■ " " 208 , ,52.4 

Private ' ^ , 184 . ' 46.3 

,1 NCATE accredited ' 253 63.7 ' 

AACTE member ' ' • -328 8?.6 - 
.Regional Accrediting Assoc. 

Hsyt England ' ' 17 , , 4.4 ' 

Middle States ■ • . , 45 ' " •1>.5 ■ 

North -Central ^ , 192 48.4 

Northwest ^0 5.1 

Southern ' ' 107 27.4 

Western ^ , 9 . 2.3 
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TA?LE 2 

Surrsnary of Institutions Responding by Size of Enrollrnent 



Enrol Iment 



Total Inst. 



300 to 1 .300 1 ,400 to 3,600 3,779 to 6,500 6,808 to 10.000 10,0^6 to 20,000 20,1j2 to &3,115 



% 25.6 ^ 24,4 16.6 ^ 15.0 . , 7.3 

Zm % ^ ' 25.6 50.0 ' 66.6 ' 77.7 92,7 ^ lOO.O 

Under Grad. ^ ' - ^ ^ 

Teacher Ed. , . . , ' 

Program 13 to 100 M04 to 250 ^ ■ 255 to 500 ' 525 to 1 ,000 1 ,100 to 2,000 ' 1,^11 .to 7.300 

t 14i,0 ^ 2KV^ 22.6 19.7 . \ 15,8 ^ 6.8 

Cum % K,0' - 35J 57,7 ^ 77,4 ' 93.2 ^ . 1^00,0 



~- ^ — \ ^ 

Ji^alid i;ases - 386 , , ^ " ^ 



s 



TABLE 3 



.^^iinrrtary^Analysfs of Institutions UtilfZ'ing ^ 
Questiorroaires fn'Evaluation-of Teacher Education Programs 



Data Source 



dept. supervisor, etc) 
Peer Teachers 

^ ' ,Pupi Is. 

* 

' Parents 
Other 



r 
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Questiofsnai res 



Up Year 





Entry 


' Durtng 


Exi t ' 


< 1 


. 2 


3 


4-6 


7-10 


11 + 




119 


169 


223. 


180 


71 


96 * 


96 


12 


5 




35.1 


49,9 


65.8 


' 53.1 


20.9 ■ 


28,3 


,28,3 


3.5 ' 


1,5 


f 


57 


145 


J53 


151 


51 


n 


.66 


7 ' 


4 


% 


16,8 


42.3 


45,1 


44,5 ■ 


>5* 


20.9 


19.5 


2.1 


1,2 


f 


10 


53 


38 


■ 11 


'3 . 


7 


7 


0 


0 


% 


2,9 


15,6 


11,2 


3.2 ' 


.,9 


2.1 

< 


2,1 


0 


0. ■ 


f 


4 


- 26 


13 




8.' 


4 


6 


5 . 


0 


1 


% 




.,7.7 


3,8- 




1.2, 




1.5 


0 


,3 


f 


, 2. 


5 


, 3 , 


0 • 


0 


- 0 


0 


0' 


0 


■% 


,6 


1,5 


' ,9 


0 


■ 0 


0 


0 


0 


0 


f 


7 . 


r\- 


"12 ■ 




1 


2 


^0 


0 


0 


% 


2,1 


2t7 


■3,5 


2,1 


.3 


. ,6 


0 


0 ' 


•0 



TABLE 4 



. SurtHuary Analysis of Institutions Utilizing 
Personal Interviews in Evaluation of Teacher Education Programs 



Personal Interviews 
Preservice Program 



Fol low Up Year 



Data Source 

The student/ ^ 

graduate ^ 

A Supervisor ^ 

(adv.isor, principal , ^ 
dept, supervisor* etc:) 



Peer Teachers 
Pupils 
Parents 
other 



f 
f 
f 
f 



Entry. 


During 


Exit 


.1 


2 


3 


4-6 


7-10 • 


11+ 


88 
26 


105 
31 


87 

25.7 


32 
9.4 


2.9 , 


9 

2.7 




^s 


3 

.9 


37 

10.9. 


91 

26.8' 


84 

24.8 


35 

10.3 


.12 
3.5 


12 
3.5 


ler 
4.7 


2 

.6 


3 

^9 


6 

1.7 


35, 
10,3 


26 ^ 
1j 


6 . 
1,7 


1 

.3 


2 

.6 


1 

,3 


1 

" ,3 


0 
0 


3 ■ 
.9 


10 ^ 
2,9 


5 . 

. ■ 


2 

,6 


2 

,6 


1 

,3 


> 0 
0 


0 
0 


0 
0 


1 

, .3 


3 

v9 


3' 
.9 


2 

.6 


2 

.6 


1 


0 

0' 


0 • 
0 


0 
0 


' 2 
• ,6 


' ' 6 
1,7 


3 

- ,9 


1- 

.3 


0 
,0 


0 
0 


■ 0 
0 


0 
0 


0 
0 


« 
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TABLE 5 

Summary Analysis flf Institutions Utilizing/ - 
Direct CjJa&'sroQm Obs^\iations in Evaluation 
of Teacher Education Programs 

Direct Classroom Observations 

Preservice Program Follow Up Y^ar 



Data Source 

The studen^^ . 
graduate. ^ 


Entry 


During 


Exit 


. 1 




3 ' 


4-6 


7-10. 


11 + 


55' 
16.2 


225 J 
66-4' 


. 139 
41.0 




34 

10.0 


12 
3.5 


11 

34 


3.2 


3 

.9 


2 

.6 


Pupils ^ 


9 

2.1 


■42 
*12.4 


29' 
8.6 




12 
3.5 


6 

1.8 


5 ♦ 

1.5 


4 

1,2 


1 

.3 


1 

..3 


Other ! 


1 

2.1 


. 28 
8.3 


19 
5.6 % 


r 


'5 
1-5 


1 ■ 
.3 


0 

0 - 


2 

.6 


0 ■' 

0 


0 
0 


4 




4 










^ 1 


/ 





TA^LE 6 



^^rty Summary Analysis of Institutions Utilizing-^ 
>^ ^^edia-Aided Work Sample ^Bbservati ons in Evaluation 
„ of Teacher Edycation Programs 



Data Source 

The student/ 
graduate ^ 

Pupils ^ 

i 

- Other 



Media-Aided Worlctfiample Observations 
Preservice Program . 



FoUov/ Up Year 





Entry 


Duri ng 


Exit 


1 




2 


3 


4-6 


7-10 


■ 11+- 


f , 




158 


"33 


5 




1 


3 


0 


0 






4.7 




9.7 


1. 


5 


.3 


.9 


0 


0 


..3 




3 . 


^ — - 

22 


9 


2 




.0 


1 


0 


0 


0 


% 




6.5 


■ 2.7 




6' 


■ 0 


.3 


0 


p 


■ 0 


f 


0 , 


8 




0 




0 


0 


0 


0 . 


0^ 


% 


0 


2.4/' 


N.3 - 


0 




. 0 


0 


0 


0 


0 
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TABLE 7 



SuTwndry Analysis of Institutions Utilizing 
Standardized Tests in Evaluation'.of^Teacher Educ^ation Programs 



Data Source 

The student/ 
* graduate ^ 



Other 



Standa^rdi zed Tests 



Follow Up Year 





Entry 


Duri ng . 


Exi t 


".1 


2 


3 


4-5. 


7-10 


n + 


f 


46- 


28 


99 


e 


5 


6 


3 


0 


0 


% 


13.6 . 




29.2 

* 


2.4 


K5 


K8 


.9 


0 , 


0 


f 


0 


2 


t 

3 


■ 0 ■ 


0 , 


0 


*0 


0 


0 


% 


0 


.6 


. '9 


0 




0 


0 


0 


0 


f 


0 


. 0 


* 0 


3 V 


0 


.0 




0 




% 


' 0 


0 


0 




0* 


0 


0 


0 


• 0 
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' ^ Summary Analysis of Institutions Utilizing 

P pfifessional .Competence \n^ E^raluation of Teacher Eduration Programs 

Professional Competence 



Follow Up Year 



Data Source 

The student/ 
graduate ^ 



A Supervisor 
(advison, principal , 
dept. supervisor, etc J 

.Peer Teachers 



Barents 



Other % 





Entrji. 


Duri ng 


i Exit 


1 ■ 


2 


3 


4-6 


7-10 


11 + 


f 


47 


138* 


154 


95 


30 


43 , 


39 


3 


2 


1^ 


13.9 


40.7 


: 45.5 


28.0 


8.8 


12.7' 


11.5 


,9 


.6 


f 


38 


151 


147 


107 


29 


45 


43 


3 


1 


% 


■ 11.2 


44.5 


43.4 


31.6' 


8.6 


13.3" 


12.7 


.9 


.3 


f 


'■12 


42 


34 


13 


2 


10 


4 


0 


0 


-% 


3.5 


12.4 


.10.0 


,3.8 


.6 


2.9 


1.2 


0 


0 


f 


5 


15 


15 


10 


4' 


8 


5 . 


0 


0 




1,5 


■ 4.4 


'4.4 


2.9 


. 1.2 


2.4 


■1.5 


0 


0 


f 


2 


4 


2 


■ 1 


1 


1 


0 


0 


a 


%' 


• .6 


1.2 


.6 


.3 


.3 


.3 


q - 


0 • 


0 


f 


2 


9 


9 


V 

2 ' 


0 


1 


0 


0 


0 


% 


.6 


2.7 


2.7 


.6 

1 


0 


.3 


0 


0 


0 



TABLE 9 



Summary Analysis of Institutions Utilizing ' 
Personal Characteristics and Attitudes in Evaljjation 
of Teacher^ Education Programs 



Personal Characteristi{;s 
Preservice Program < 



Follow Up Year ' 



Oata Source^ Entry During Exi t 1 



4-6, 7^10 11+ 



The student/ 



graduate ^ 



A Supervisor 
(advisor , prinli pal , 
dept, supervisor, etc.) 

Peer Teachers 



Pupils ^ 



L Parents-fc 

ather 
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71 

20.9 


126 
37.2 


129 

'•38.1 - 


65 

19,? 


17 
5.0 


33 

19.7 


28 
8.3 


4 

1.2 


2 

.6 


53 

15.6 


127 
37.5 


115 
33.9 


75 
22.1 


17 
5.0 


34 

10.0 


F 

33 
9>7 


3 

.9 


1 

.3 


13 
3,8 


41 

12J 


--'30 
8,8 


9 

2.7 




2 

,6 


4 

1.2 


1 

.3 


0 

0 • 


0 
0 


3 

.9 


9 

2.7 


12. 

3,5 


8 

2.4 


3 

,9 


6 

1.8 


4 

1.2 


0 

.0 


0 
0 


2 

.6 


4 - 
1,2 


2 

.6 


1 

.3 


1 

.3 


1 

.3 


0 
0 


0 
0 


0 
0 


7 

2.1 


11 
3.2 


6 

1,8 


2 ■ 


0 

0 


0 
0 


0 
0 


0 , 
0 


0 
0 
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\ WHAT NOT TO 00 IN EVALUATING TEACHER EDUCATION PROGAMS ^ 

DOZEN DON'TS 



Lyn Gubser - ^ 

National Council for Accreduation of Teacher Education 

! ■ ■ •■ 

Se^ng someone who tries to take a positive approach to thi^ngs and who 
enjoys k natural tendency to rebel against lists of "don*ts/ I was a bit pift 
off bx He request that I begin oun discussion of teacher education eva?tjation 
by descpbing what not -to do,y I find it saddening to study program evaluation 
effortSj^ that have failed to fly--cr ashing and burning after takeoff, )f they 
even gcjt off the ground/ But if our task is Tto e)(amine the wreckage of failed 
effort^, then at least let conduct our investigation with^'an eye out for 

clues to future success. 

/ 

The Lay of the Land 

Evaluation of teacher^ducation Involves the appraisal of the entirety of 
any system thai; prepares schooTpersonnel, f* includes the assessment of pro- 
grams that prepare counselors* school psychology sts» administrators; media 
.tectyiologists, librarians, and assorted specialists, irt addition to instruc- 
tio/ial personnel , In planning for the evaluation of prjDfessional programs, 
on^" should npt overlook the e)(tensfvely broait range of specialties that pro- 
fe^fisional education has come to include, ^ 

1 If we regard programs as systems, then we must be concerned with the 
^valuation of inputs, processes, and outputs. Attention should be paid to the 
evahjatlon of such factors as program admission and retention, of clinical 
facilities and resourC^, of faculty competence .and experience. In addition 
to the assessment of qualities of those who complete professional programs. 
One of the most common malpractices In 'teacher education evaluation, as. we 
shall see, is the frequent confinement of assessment efforts to the evaluation 
of graduates. 

Sound program evaluattfions can be constructed on any number of philosoph- 
ical and Contextual foundations* Program review' is one approach and is^ome- 
what analogous to calling together a group of chefs to evaluate the baking of 
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*a cake, Frotn^ their combined knowledge , and experience decisions are made on 
the amount of flour, sugar, and flavorings to add to the recipe to obtdlin the 
desired result. But decisions must also be made on process variables* such as 
how hot one should keep the oven, Quality control is a continuous procedure 
in baking. Although the ultimate question is how tasty is the bake, a 
multitude of evaluations' are made before that question can be asked. Even 
then there remain such , long-range questions as how long can the cake be kept, 
Uftder wha^*^conditions, and how best should it be served? 

Me in education have tended to focus on what goes into the cake. All of 
us have had our pet suggestions* most of which have been unsupported* by Vel 1 - 
able dat*. Only within the past decade have we actually tried to "taste the 
cake." For many reasons our successes- have been infrequent, 

Educators have only rarely sought to7 assess in-put variables or evaluate 
processes, in empirical «ays. Few evaluation models have attempted to system- 
atically assess textbooks or establish a sequence of program content through 
careful investigation. We are hard-pressed to comment upon the- successes or 
failures of these evaluation approaches, for one cannot effectively criticize 
that which only rarely exists. If enrollments are up and the central admin- 
istration is happy* that's been good enough for many of us, and we have pushed 
student te.achers out of the academic nest and iriti\^the classroom. If they 
haven *t returOed, we assume that they flew away to bright careers. 8ut we 
really don't know whether they succeeded or failed, or whether the program had 
anything to do, with this success, or failure. Without continuous or appropri* 
ate* evaluation we go from one whim to another without knowing whether the pro- 
ducts of our earlier efforts were highly successful or complete failures. 

Even when softie components of an evaluation system are operative* others 
may break down,* We recently observed a spl^enjjid program effort in evaluating 
the student teaching program of a major un'lversity. This university placed 
70% .of its student teachers in school systems remote from campus. Resident 
university supervisors were employed to assess student teacher peformance to 
determine the strengths and weaknesses of specific programs through^ measures 
of student teaching 'competency. They employed interaction -analyses, pre-» 
mid-» and post'-exqerlence inventories , surveys of students and cooperating 
teachers 'an3 administrators* exten,sive observational data* diagnostic instru- 
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merits* cogn.i'tive level assessments* and a host of other devices. But then we 
found that none of this information was ever referred back to the teacher edu- 
cation departments! Just imagine that! Once students had cofnpleted their 
work in subject matter or pre-clinical specialties, their respective depart- 
ments ne^r heard from them again. This was the only evaluation effort the 
institution employed, yet when faculty were questioned about the program they 
bragged of its quality, citing as evidence the small number of informal com- 
plaints received about their products. Even the .SujJervisors drew unfounded 
conclusions regarding the academk program. They assumed that if student 
teachers were successful, the academic program must be functioning at an opti- 
mal level. Because this institution served a wealthy and well-educated clien- 
tele» however, it "was just as plausible that graduates were successful in 
spite of their experience in the program. Or perhaps their skills could have 
been improved.' We will never know. 

. ' I owe to James Raths of the University of Illinois a debt of gratitude 
for pointing out another mountain against /which evaluation efforts are f^^e- 
quently dashed. Professor Raths notes^at teacher educators frequently get 
trapped into assuming that the assessment of individual ' teaching competencies 
is the essence of program evaluation. Says Raths: 

It is a matter of getting involved in an infinite regress. If you 
respond by saying we want good programs to have good teachers, I 
could ask why we want good teachers. The answer is that we want to 
have good products gradpate from our schools. If we pursue t1iis 
questioning, we could ask why we want outstanding school graduates; 
the answer is to have a better society* Why do we want a better 
society? So that,,*where does it end? 

Raths* best guess is that "...each of us has an arbitrary focus on an infinite 
chain." The focus "of program evaluation, he believes* '^is properly on the ^ 
quality of programs and not on the quality of teachers/** Teacher educa- 
tors, however* are often pressed to use program .evaluation In lieu of the as- 
sessment of individual skills. This confusion of apples and bananas has been 
greatly abetted,^ and even enshrined irjto law» by state program approval ef- 



*James Raths. Personal^ correspondence of February 25» 1980. 
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forti. Many state departments of education have found a cheap way to license 
teachers by handing a certificate to anyone having completed an "approved" or 
accredited program. Yet no model of program approval or accreditation can 
possibly determine the competencies of individual products of those programs. 
U is not a task appropriate to program evaluation, nor for that matter to 
najtional accreditation, although we frequently are criticized for not doing 
'it. Even if we could, the impact of school systems upon ^the behaviors of . 
individual teachers probably would overshadow program effects within a few 
short years* Programs ar^ designed as \ehicl^s for providing opportunities 
for behavioral change. There 'can be no guarantee that these changes will last 
or will not be altered through subsequent experience. 

The Evaluation of Graduates 

Nothing herein is meant to detract from the importance of eva'luating ttie^ 
products of Our programs* This is not to say, however, that program evalua- 
tion should rest exclusively on the evaluation of graduates. Nor should we 
assume that evaluation *^^f graduates and assessment of individual competencies 
have the same purposes or ^ims* i 

Because so much program evaluation has focused on the evaluation of grad- 
uates, there has accumulated a significant amount of information on the sub- 
ject* We find^ of course^ that in the area of graduate assessment have oc- 
curred our greatest number of mishaps and total disasters * Use of program 
evaluation in an attempt ' to guarantee individual competency has resulted in 
many institutions attempting to evaluate the entire population of their grad- 
uates, with an almost universal lack of success, 

NCATE published in 1970 the first revision of its standards that required 
institutions to evaluate their graduates and use the results of these evalu- 
ations for program modification and long-range planning* 'The Council's intent 
was to complete the loop of a system by providing feedback oo program output 
in conjunction with other evaluation data that could be used to modify input 
and process variables* Wenow have a ten-year history of efforts to evaluate 
programs^ providing a reasonable basis for drawing some conclusions about the 
efficacy of teacher education evaluation on a national scale. We regret to 
report that these efforts have been less than impressi^ve. # 



For the past two years violations of standards of program evaluation have 
constituted the top two reasons for denial of NCATE accreditation for both 
basic and advanced programs* Even where denials did not occur, lack of' 
effective evaluat ion systems led the li st of weaknesses cited for 
baccalaureate and graduate programs at 143 institutions reviewed. 

Where programs failed there generally was no lack of motivation* Evalua- 
tion of graduates and other aspects of progam assessment have J)ecome the most 
common topics at seminars, meetings, orientation sessions, and anywhere ^Ise 
accreditation concerns are raised. We recognize that in' some instances tech- 
nology of measurement simply has not risen to a level consistent with profes- 
sional expectation, Yet with that concession notwithstanding, we have observ- 
ed many common pitfalls that almost guarantee the failure of efforts to evalu* 
ate graduates. 

Institutions ^hat concentrate their evaluation activity on the assessment 
of graduates frequently tack this evaluation on to preparation programs almosV 
as an afterthought. Rather than construct the concept of evaluation of gradu* 
ates into the general evaluation system of a program, many institutions pay 
little attention to evalu'ation until they begin to prepare for an NCATE evalu* 
ation or other program review. Some institutions regard program evaluation 
and national accreditation as one and the same thing, (One result of this has 
been that institu'tions attribute great Cost to NCATE accreditation because 
they add every nickel and dime spent on evaluation to the total accreditation 
bill. The difficulty in .countering this claim is that the only reason many 
schools do conduct evaluations is because of accreditation. Few successful 
program evaluations are. conducted when the only motivation is to "pass profes- 
sional muster,'*) * , ^ * 

The least productive approach to program evaluation has been that of 
evaluatin^^raduates through attempted surveys of entire graduating classes^. 
Institutions have demonstrated ^ fundamental lack of Ability to track gradu- 
ates and sustain contact with them, This has resulted in survey returns of 
403£ beiag considered phenomenal , with normal returns frequently running at 
less than 153£. Low return rates breed problems of sampling. One must ask who 
are the respondents? Those who are unemployed, angry at their former inst'itu* 
tion? Those who have enough time to write nasty responses? Or are they the 
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extremely dedicated who enjoyed their program and would not criticise it ^even 
constructively? . ' ^ 

What questions can one asR that will provide useful information?. Surveys 
must measure what they can, not what may actually be there. The result is 
that graduates are aske/d to respond to mundane and nebulous questions: "What 
did you feel was the most Important aspect of your training program?" "What 
did you feel you missed in your preparation program?" Answers to such ques- 
tions are pred|ctable and less than helpful , Individuals respond' that the 
most important aspect of their program was student teaching. They usually in*r 
dicate that they needed work in discipline, These responses tend to occur 
even when individuals have had inordinate amounts of student teaching and when 
classroom manage*ment and discipline, were primary components of program exper- 
ience. We find that surveys ofterf measure not the knowledge and skiDs stu- 
dents acquired in the program, but what common wisdom tells them to think they 
acquired, or did not acquire* Surveys tell us' more about peer knowledge and 
group methodology than they do about preparation programs completed by pro- 
spective educators. 

Surveys of employers have -also proved less th,an helpful* One midwestern 
institution rec^tly became suspicious when for the third consecutive year 
pertain employees of their graduates responded with almost identical comments 
of praise for program graduates. Neighboring institutions were therefore con-, 
tacted to determing the nature of responses they were receiving from these, 
same employers. Some were amused to find that the same principals and supj&r- 
iptendents who had indicated they would hire no graduates other than thoie 
of a particular institution, had told competing institutions the very same 
thing. Although exceptions do occur, in general employers seemed unwillfng to 
be candid about the quality of personnel they hire at least when responding to 
mailouts* Perhaps to be critical would be to criticize their own selection 
and employment procedures. Some, may actually fear legal reprisals from cr;'iti- 
cized employees, - ^ _ * ^ 

Anpther common problem with surveys is that they attract information that 
IS not current* Many schools report that when they do receive constructive 
information from respondents, it is outdated* Former students criticize pro- 



fessors who have not been ernploy^d for years and courses that rio longer ire in 
the curriculum. They typically advise that courseWork in reading and special 
education should be added when it probably becibe a part of the curriculum 
ye^rs ago. Survey techniques tend to ask simple questions and^ evoke simple 
responses--gut leveJ reactions that may be no more timely than they are accu- 
rate, Sui^eys domot get at classroom technique, level of knowledge of the 
teaching speciality^ or anything else that one would normally list as high on 
the priority of essential knowledge and skills for educators, 

"Evaluation efforts often fail because they- are insufficiently financed. 
The adage that one gets what one pays for could be no mot e true, than 1n ref- 
erence to the assessment of graduates, A last-minute» one-page survey ques- 
tionnaire will glean about as much useful information as the time and money 
that go into such minimal efforts. The last place one wants to become stingy 
^TS in the area of evaluation. If done appropriately^ evaluation can actually 
s^ve money by making programs more cost effective^ to say nothing of improving 
general ^academic quality, 

It is not my assignment to describe successful program evaluation activi- 
ties and techniques. But with respect to the evaluation of graduates^ I wi]l 
note that NCATE has found the most successful evaluation systems to be those 
that employ sampl ing , suitain contact with a limited number of caref ul ly- 
selected, representative .graduates, and that place evaluators^ in the .field, 
rather than rely exclusively upon surveys. All too frequently^ institutions 
completely overlook such^ face-to-face techniques as interviews of graduates^ 
their employers and students. Sampling the opinions of a diverse population 
of graduates » their peers and employers » can provide terri f ic insights into 
program needs, not only for pre-service preparation* ^but for in-service pro- 
gramming as well. Many institutions overlook the potential of internships and 
other residency exper^fences thai could provide opportunities for acquiring in- 
valuable feedback onithe performance and capabilities of graduates and those 
who supervise them, Argyris and Schon (1977) have demonstrated that some 
residency or internship for teachers and school support personnel would go far 
in providing invaluable data on program straxigths and weaknesses/* 



. *Chris Argyris and Donald Schon^ Theory in Practice: Increasing 
Professional Preparation ^ New York: Jossey-Bass; 1977. 
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' Using Results of Prpgram Evaluation . ' . * * " 

" As common as any problem with evaluatiron efforts has been ^a general 1n- 
/ clination of prof essiorval faculties to ignore results of prograrrf evaluations, ^ 

Evan 1n some natfionally^acclaimed effort^, ^ scant yse of information resulting 
^ from these evaluations' has been made by resident faculty^ One cannot expect 
merely because an evaluation effort bears fruit', that teacher educators -wi U 
beat down the door to acquire criticisms that might suggest need for program 
modification and change, reordering of departmental priorities; or--Sod helo 

us--r6assignment of faculty- ■ . . " 

■ if ^ * 

If an evaluatioa program is, in the final analysis, to be ultimately suc- 
cessful in modifying faculty behavior, then from the outset it should^ involve ■ 
program fatuity in artict>4^4ting evaluation efforts wi th program * aims and ^ ^ 
objectives-. To do less is to guarantee failure of the' evaluation mission. 

Again, program evaluation -should be comprehensive, examining the curricu- 
lum, procedures, and processes that affect student admission, courseware, 
clinical and field experiences, as we>l as the evaluation of graduates. In- , 
^s^itutipns that fail to assess the function or^u^fity of any of these compo- 
nentS clearly limit not only the success of tKeir evaluation effort, but also 
severely restrict the probability of academic syccess/ 

Summary . - 

To summarize, "here are 12 suggestions of what n ot to do vf one aspires to 
*es^ablish a successful program evaluation effort; ^ 

A DOZEN OON'TS . - . 

1, OON'T |ail to evaluate program entrance, variables, including admission 
criteria, counseling procedures, and other factors that may affect opera- 
tions throughout the entire prograpr,; 

2, DON'T- fail to evaluate the criteria and procedures of. program retention, 
incliwirtg assessing the effectiveness of scr^ning^mjteria, procedures^ \ 
for counseling,<)ut the unsuited, and fdr determining remedial procedures / 
for tho^e whoSe performance isjninimaK 

ft 

3* OON*T* expect evaluation of professional programs to result in tight sta- . 
tistlcal data that can b6 interpreted with little judgment. Professional 
decisions are- subjective ontfs. 
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4. DDN'T place all your eggs in one evaluation basket* Diversify. Provide 
checks ,and balances, for the testing of validity throughout the program, 

DON'T expect, program evaluation efforts to be\uccessful if conducted on* 
ly sporadically. Evaluation should be an Integral part of each aspect of 
the program from planning to execution, from .admission to graduations 

6. DON'T try to get by' on the cheap, You get wha^ you pay for In program 
evaluation. Evaluation efforts usually are dost effective. Their pri- 
mary value is that they spotlight areas in which fiscal and personal re- 
^sources^ are being wasted, as well as those components that are function- 
ing efficiently. . 

7,. " DDN'T fall into the trap of assuming that the assessment of^TRe^ndi* 
vidudl skills of graduates is equivalent to^the evaluation of pragrams, 

, 8. DON*T bother with correspondence^surveys of employers or graduates, un- 
less this is the final resort, ^ ^ 

9. DDN'T fail to use interv,iew t^diniques, Concentrate opportunities af- 
forded by practica and internsn^ for ; acquiring face-to-face reactions 
to questions re^rdlng program operation, 

' 10^, DDN'T Jffhore important evaluation feedback even if it is critical and — 
politically unpopular ia the short-run* Data supporting negative conno- 
^tations can best be handled when kept in the-context of long-range plan- 
ninq< 

\ ' , 

DDN'T assume that just because evaluation data are available that indl* 
viduals will automatically use it to modify and improve programs, A 
K:cihscious effort must be mounted to school faculty and administrators in- 
to employing evaluation dat'a in planning anci program modificatfon, 

IZ. DON'T treat evaluation as an afterthought or. conduct program assessment 
becOTse spme state department or accrediting agency requires it, An 
evaluation should be as much a built-in component of professional pro- 
, grams as Vurricula, clinical; experiences, or any other fundaroental part. 
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USE-ORIENTED EVALUATION 



• James Craig 
**RonaTd D, Adams 
Western Kentucky University 

INTRODUCTION - 

Evaluation of teacher education programs has been a topic of intense 
discussion both inside and ^outside the educational community in the last 
decade* The focus to a, large degree'has centered on practical considerations 
related to data collect'ion and analysis methodologies appropri ate for use in 
assessing the effectiveness x>f special public school programs, federally ^up- 
potted projects, and the like (e,g,, the classic Campbell and Stanley (19631 
text}* ^lTo"durTng" this period of time, a"great deal of journal ^paqe has 
been devot^ to the theoretical aspects of evaluation, in teacher .education, 
such as formative 'and^^^ative evaluation (Scriven, 1967) and goal -free 
evaluation -Striven, 1972), as well as. modets of the evaluation process 
(PophOT, ^972; Wortmart, 1975; Sti^ff leb^am, 1978), In all instances, the 
emphasis has primarily been on method, - ' ■ . 

In the past decade, however, many individual^ involved in evaj^uation 
have become increasingly^ aware that evd'luation is not "method bound" but 
ralher is bound b/ the social/political context within which evaluations are 
conducted and data are used (Weiss, 1972; Suchman, 1972; Mathis, 1980), 
Ideally, evaluation dat^a are used to make objective program decisions regard- 
ing program development, modification, operation and ttie like. However, it 
often seemi rare that eval on data are used irv such' a direct, straight- 
forward manner. Therefore, one of the primary evaluation Concerns in recent 
yeai^s has been the development of ways to obtain more effective use of evalu- 
atiofi data in program decision making' {e,g, , Patton, 1978; Mathis, 1980). 
The important term is use. 

The definition , of use typically advanced is that use occurs when evalua- 
tion data are directly employed in objective, observable ways in program mod- 
ification and operation (Cohen, 1977; Patton, 1978^; Mathis, 1980) However , 
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to such a definition does not fully incorporate the social/political context m 
which programs operate and the fact that -evaluation data constitute just one 
of several bits of '^rfrormation that entered into program decision making in 
teacher education (e.g,,^ governmental statutes and personal gain), A more 
realistic view is that use of evaluation data in program decision (T^^iklng is 
^ almost always diffusa ^n nature and not always (if ever) directly observable 

(Patton, 1978)^ use iS an iterative process which^focuses on the assimilation 
of evaluation ^Information in^o the decision making process. 

How then 'can evaluation systems be structured to better use the data 
which th^ systems provide? We believe the answer to that question is to 
develop evaluation sj{stems for use In teacher education where the emphasis 
from tlie very beginning on building mechanisms to increase the systematic 
inclusion of evaluation data in pr9gram decision making! The evaluation sys- 
tem we intend to present foV y«<r consideration focuses on use and was de- 

*. 

veloped out of both* our own .attempts to conduct evaluations of programs and 
, in trying to assist others. Many J&f the idSas are not new; most, if not all, 
have been directly (Suggested.or ^t^east, anticipated in one way or another by 
others who are actively involved in evaluation. What is new is the enphas^s 
on use and the manner in which .that emphasis is incorporated into teacher ed- 
ucation evaluation systems. * ■ 

, y ' BASIC PRSMfSES 

Our proposal s ^rel^basedi on the following four premises: 
K The purpose for .establishing an evaluation system for a teacher edu- 
cation program 1s t6 provide r^elevantf meaningful information to be systemat- 
* * * ' 

ically incorporated into pfpgLK*^] decll slon making . Establishing an evaluation 

'■system simply tQ have one^'/lnjifface for an NCATE visit cannot and should not 
provide the sole motivation^ The NCATE standards are well intended, but it 
remains for us, the prof eSsJ.9^al teacher education conmiunity, to give these 
standards meaning and va^ue — especial ly ' regarding evaluation of our own 

^programs, ^ J 

2; All individuals respot>sible for program implementation and operation 
should be identified- VTKe organizational structure within which most teacher 
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education pifograms operate and are evaluated can be represented by the hier- 



archy shown in Figure 1, The chief administrative officer (usually a dean or 

department head) ' is the individual responsible for establishing the^ 

parameters (e.g, , the allocation of resources) within- which programs must 

Operate. Also co^ained within the iypical organizational structure ' is a 

program administrator .(e.g.,, a department head or a coordinator) charged with 

operating the progl^am, a program staff (e.g, , a departmental faculty) 

responsible for delivering program services, and program participants (e,g,, 

undergraduate students) who are the individuals who receive the services the 
^ , 

program was created' to ^-ender, : , 

3, If evaluation dataJare-'to be effectively used in program decision 
making, then individuals at all levels of a program must be irfvolved in the 
evaluation effort. This riteans that not only program administrators and pro- 
,gram staff must be include^ in the development and operation of the evalua- 
tion System, but that adfTr?hi$trators higher in the organizational structure 
must also actively participate in the evaluation process, incorporating the 
evaluation data Jnto , their own (lecision making regarding programs and related 
pol icy concerns . 

4* The e^/aluation of teacher educaticm programs must operate within the 
organizational framework which Cijrrently exi sts. Furthermore^, evaluation 
should be a part of the ongoing operation of the program; it should not be 
perceived as a "lay-on*' by administrators, legislators, or other "outsiders/* 
In addition, it should be realized at the outset^that it is very unlikely 
that new monies will be made available to ^fund program evaluation efforts. 
Evaluation will have to occur within-the context of the operation of on-going 
teacher education programs, planned professional development efforts , and 



related activities. 



USE-ORIENTEO EVALUATION 



The use-oriented -evaluation system we propose is composed of five com- 
ponents (see Figure 2); Initiation and Verification, Planning, Implementa- 
tion, Data Collection, and Feedback and Utilizatl^. These Components are 
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each discussed ^below wUh the focus being what can be done to increase the 



Initiation and Verification 

A summary .of the organizational chart as it reflects the activities at 
both the Initiation and Verification and Planning stages is presented in 
Figure 3, 

^The impetus for implementing an evaluation system may come from outside 
or from within the organization. Regardless of the source or the position of 
the initiator, the Initiation and Verification phase of an evaluation ulti- 
mately must begin with the chief administrative officer respons'ible for^ the 
program un it" -usual ly a dean . Without that individual *s understanding of, 
commitment to, and approval for the evaluation of a pro'gram, the evaluation 
effort will not be successful. 

Individually (or perhaps with the assistance of an experienced evalua- 
tor)» the chief administr'ative officer should conduct a preliminary overview 
of the Evaluation and an analysis of its various rami ficat ions . The infej^t 
is to identify the chief administrative officer's perceptions of the present 
institutional and program circumstances* the ideal progf^am, the need to con- 
duct an evaluation* the possible options for implementing the evaluation, the 
possible program implications that might be .suggested by different evaluatton 
data,. and the resource restrictions within which the evaluation must be con- 
ducted. The objective is NOT to have the chief administrative officer estab- 
lish the form and substance of what the evaluation should be, but rather to 
provide her/him with an understanding^ of how the evaluation process may oper- 



\ ' ate. This process wi 1 ) aHlow the administrator to reali st'ical ly set the 
parameters wi thin which "xhe evaluation must function (e.g . » the budget ), 
develop a conwnitment to the evaluation process* and identify possible program 
decisions which could result. 

If, at that point, the decision is stil) to institute an evaluation sys- 
tem» then the preliminary review should be repeated^^^fTfFT those individuals 
within the. organi zation who servey-«5" the primary decision makers for the pro- 
^ grams to be evaluated. The decision making-team should be identified by the 
chief administrative officer. It should be composed of individuals who col* 



frequency of use of evaluation information 'in program decision 
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lectively facilitate the establishment of specifid program goals, functions, 
and operational procedures, as wel 1 as those who are respons ible ,for 
decisions regarding program modi fication and continuance ,(e,g -an assistant 
dean, a department head, and a43rogram coordinator), Tjie same cansiderations 
addressed by the chief administrative officer should be addressed ,by the 
decision making team. In addition, this group should determine th^ 
procedures for the creation of the planning and evaluation team, 

Planning • 

The planning and evaljatio'n team u the primary initiation and opera- 
tional force m an evaluation effort. The team determines the details of the 
evaluation (e.g., the types of data to be collected) and is responsible for 
^cofwnunicating to all concerned the form and substance of the various evalua- 
tion actwUies, So that the team can function effectively, it should be 
composed of both fonnaj^^d informal program leaders and b^ limited to a nax-' 
imun of ten members, The exact composition of the team will be specific: to 
each situation and determined by suc^ th^^ys as the number of program 5taff, 
the leadership style of the chief administrative ^officer, budget restric- 
tions, ao6 similar' factors. 

The planning and evaluation team has four charges. First, it should 
create a viable, flexible, workable evaluation plan agreeable to the team 
member^'that includes at least the following characteristics; 

a, a specification of the evaluation data required to make decisions 
regarding the present s^tatus of the program (with'in the boundaries establish- 
ed by the decision makers); ^ ^ 

b, the rated importance of {the various dat a xons istent with the teach- 
er education program goals^ and objectives and the current knowledge regarding 
effective teaching; i 

c, an identification and lasting of the possible sources of data; 

d, an evaluation of the possible data sources and collection procedures 
in light of access, jcost, developmental time, staff development needs, and 
time delays; - ^ 

e, a pr1oi*iti2ation of the jx>ss1ble d^ta ^sources and collection proce- 
dures in terms of the evaluation .data required and related resource restric 
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tionS; 

f, and, a description of the available data collection procedures se- 
"lected and/or Mapted, new p^ocedures developed, and the data col lection 
training reqi^red '(if any), - 

The second charge ofjathe planning and evaluation team should be to 
create an ef fecti ve» operational cormiunicat ion system* The conimunicat ion 
system should" be used during the development of the evaluation plan to ensure 
the involvement of all individuals in planning and wdifying the evaluation 
system. If" implemented properly, the system\should function after the evalu- 
atien plan is operational to disseminate evaluation data to al 1 concerned 
faculty and administrators for use in program decision making. 

The third charge of the team sWould be to designate someone to be the 
evaluation mapaqer* Space does not permit us to fully elaborate the role of 
the evalu£ttion manager. At this point U sufficient to say th^t the eval- 
uation manager should se»tve as the tearp's executive officer and be respon- 
sible for the "nuts and bolts" implementation and operation of the evaluation 
plan. The person probably should have the needed tec^mrca^ skills -(although 
these could be acquired through training) and sftot/ld be a cap,able .group 
f aci 1 itator , 

Jhe fourth charge is that the planning and_ evaluation team -and the eval- 
uation manager should identify and .establish an advisory^ group consisting of 
field based educators. Input from the practicing arm of the teaching profes- 
sion is^necessary to gain the perspective of the school administrators who 
vfiU ultimately employ graduates and the perspective of the practicing teach* 
,€rs who can provide expertise gained from on-the-job experience. Others who^ 
may be important members of an advisory team would be university educators in ^ 
. areas other than teacher education (e *g. , content, areas and data pro- 
cessing). 



Imprlementation ^nd Data Collection 

Implementation and Data Collection reflect the activities associated 

* 

with the operation of the evaluation plan. Briefly, the evaluation plan 
established by the planning and evaluation team should be specified to the 
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extent that procedures for data collection are established^o include 
selection of instrumentation and data collectors, procedures for selecting 
participants, and identification of data management systems. The evaluation 
Plan is^ then implemented and ^operated by ^the evaluation manager under the 
direction of the'planning and evaluation team/ 

' Feedback and Utilization 

One of the ^most important aspects of the evaluation system is the 
development 0/ an active, effective communication system that is continually 
operative from the very beginnings of the evaluation effort. It *is^ extremely 
irnportant to rea^lize that the generation, dissemination, and utilization of 
evaluation data is NOT a lock-step, linear Sequence (Havelock^ & Lindquist, 
1980] . As was noted earlier, evaluation of teacher education programs 
conducted within a social/political environment. Therefore, the key to the 
successful operation of an evaluation program is continual involvement of al) 
individuals concerned through an ongoing exchange of ideas and values coupled 
wuh the evolution of a commitment to make the. educational program the best 

■ U can possibly be. The form that any Particular cormiunicat ion system may 

assume in-a given instance will be peculiar to tha^^ituation. Regardless of 

*the particular system that evolves, the emphasis should be on the effective 

use of evaluation data in program decision making. As evaluation data become 

available, the data sh(&u^d be systematically reviewed, assessed, and inter- 
im 

Preted by all those involved. In this regard, there 'are several important 
considerations. First, the evaluation data should be analyzed and presented 
in a form*that is both consistent with the program objectives and' readily 
' understood by the users of the information. Second, there must be a formal- 
ized process by whicfi these data are reviewed by all staff and applied in 
making program decisions. Third, the change process should be documented and 
the evaluation system reviewed in light of any changes made. 

. ^ CONCLUSIONS 

We belipve that involvement and "evolvement" are^ the keys to the 
development of evaluation systems where evaluation data are effectively used.' 

/ ' 
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First, policy makers, administrators, and program personnel at all levels 
must be actively involved. They all should participate -in the determination 
and operation of an evaluation system if it is ta be one wf^ere the data ^'^e 
validly collected and analyzed and one where the data are seri'ously reviewed 
and considered in making' programmatic decisions'. Second, it should be real- 
ized that evaluation -must be viewed as iterative in' nature where both the 
evaluation processes and products evolve over time . ^ The system shoald re- 
flect current concerns but also be flexible enough to adapt to new and chang- 
ing needs. If we want evaluation systems for our" programs to be something 
more than decorati^ ornaments, that is, we want them to provide useful 
informat ion , fov pi\ogram decision making, then we must position them ip the 
forefront so that we might adapt them for their nnost effective and efficient 
use in meeting the needs of /he e^iocation profession we col Actively 
serve. 
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DESIGN CyARACTERISTICS FOR MEANINGFUL TEACHER 

EDUCATION EVALUATION \ 

. ; 

Jerry 8, Ayers 
Tennessee Technological University 

Evaluation of teacher education program graduates has been one of the 
most, difficult prob 1 ems facing institutions of higher education during the 
past decade* Various institutions have attempted to develop models for eval- 
uating the product, ,i .e. » graduates of a teacher education, program; however, 
only a limp^ed number of institutions have implemented a comprehensive model - 

/Western Kentucky , Uni versi ty (WKU) and Tennessee Technological University 
(TTU) are^ two riristitutions that have developed , and implemented models for the 
evaluation of teacher education graduates* The models serve as a guide for 
gathering evaluative data that c^ be combined with other information for the 

, purpose of evJVuating the t&^cher education pr4)grani. The institutions have 
been employing these models/for over eight years for the purposes of curricu- 
lum development aM^^impi:aVement *The models are still incomplete and inade- 
quat^; however, much information has* been gained about evaluation of teacher 
education graduates, aod a definite set of characteristics have evolved which 
can form the basis for the deverapnient and implementation of teacher educa- 
tion fol low-i4)" evaluation (todels by other institutions. 

jgn Characteristics 




Ten major -design characteristics are essential for a rrieaningful pro* 

gran) of teacher fol low-up, evaluation* This section presents an outline of 

each of these characteristics, .some of the problems that might be encountered 

if .the characteristics are not considered, and a summary of Row the charac- 

tertistics have been made a part of the models employed in follow-up studies- 

of the^raduates of the teiacher education programs of WKU and. TTU, ^ 
/ - ^ \ 

1- The faculty, administration and governing body of the ^nstitTTTTon must 
, be committed to conducting a program ot teacher evaiuatiofl, ^ 

If an evaluation system is developed in a vacuum without a^lequate input 
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from the faculty, the riasults. of th? evaluation effort may be ^t. The, 
f^ctjlty are the primary consumers of the resuHs of a program for the evalua-, • . 
tion of teacher education graduates and if they .are to , make use of the data 
and results f^om the study, theV must have adequate input in order to arvswer 
questions they have relative to program development and improvement. Iq a 
similar manner, the administration and governing body of an instit^ition must 
be involved in t\}e project, in order for them to understated and provide the 
needed resources for conducting the evaluation program. . Without adequate 
involvement and cojimitment on the^ part of the faculty, administratlo^i and 
governing body oi an Institution, the* evaluation prograno cannot sXicceed. a 

At WKU an(^TTU, the initial ' impetus for the development and implementa- 
tion of the ftiocte.ls cafte from efforts to meet the evaluations standards of the 
Natiba^l Counc^ for Accreditation pf Teacher Educatiop/ (NCATE), However, 
the facuTty, a<Jministration, and to ^some extent the governing bodies of the 
institutiOns'were involved if) t^e program of , evaluation from the- very begin-' * 
ning. In this manner, the f acufty and administration felt the project was a 
part of their efforts in program development and improvement, - ' 

Z. A teacher evaluation program must be viewea as a part of the total 
^ teacher education program and not as an isolated project, ^ 

Industry spends a significant portion of its resources on product eval- 
u-ation, ^ AH industries i^ the competitive marketr-place have systematic, 
v^id procedures for testing how well its products perform the functions for ^ 
^hicH t*hey were designed. Institutions of higher education are only begin- 
ning to realize the value of the industrial model for the evaluation of its 
product, i,e., the graduates of its ^programs. It is realized, however, that 
the application of the industrial model is difficult ""wl^en de^aling with peo-i 
pie. Teacher follow-up eyalt^ation must be integrated i^to the total program 
of teacher, education. S^fi^^ent res^s^s must be alloc\^d to the evalua- 
tion fefXarts and they must be made a part of the total teacher educatipp pro- Q 
gram, ^ If a teacher evaluation program i,s earned out in an isolated manner, 
the results will r\ot find\their way into the total program for the, prepara- 
Hon of teachers". The evaluation efforts would thus be lost for purp^osps of 



er|c 



.50 , 



pragrdjn development %nd impro\;ement* 

WKL> and TTU have had systematic 'progranjs of teacher follow-up evaluation 
for over eiqlit^ars. At each^ institution individuals have been assigned the 
responsibility of conducting the evaluation efforts and sufficient resources 
have been allocated fo^ carrying out the eva*lLfation studies. The administra- 
tion of each of the institutions supports the evaluation efforts and have 
made every effort to provide the needed resources*' The teacher evaluation 
efforts have been built into the total program for the preparation of teach- 
eT"s, ^ ' ' ^ 

' - ' ' -. 

3, A teacher ^valuation/program must be specific/ yet reflect the total 
teacher eiJucation effort, The evaluation ettorts must retlect the goals 
and objectives of specific programs and the overall program for the 
tg^inlng of teachers. ^ ^ ^ 

* Training programs Include general knowledge which is' common for all pi^o- 
spective teaches)g» a"nd specialized knowledge which is applied according to 
the tevel or area of teaching;* Most institutions train prospective elemen- 
tary teachers/ secondary teachers iVi a variety of fields* and possibly spe- 
cial education teachers,* Evaluation efforts must include not only elements 
'commt^ to aS/l teacher training but ' to training for work in a specialized 
fjeld* For ejtamplej the training that 'prospective teachers receive in meth- 
ods of teaching, varies from elementary to secondary programs of study, TMs 

variation must b'e neflected in the evaluation program that is established* 

* 

Also* within a teacher training program there 'are certain common elements 
whi^J^ust be 'evaluated^ for^ example, basic educatloir recyuirements which in- 
cline competency in such areas 'as basic* psychology/ social foundatiorts of ed- 
Nation, and history ^nd philosophy of education* * * *. 

The" specification of objectives for the training, of teachers is an in- 
herent element in an adequate evaluation -system* If the plan for the evalu- 
ation of teachers is not based on both the specific and general objectives of 
^ the teacher education fjrogram* the 'evaluation may turn into a us^ess exerr 
^cise. The resalt^of the evaluation effort wiSuld not be tied to an^ specific 
effort that may have been made ^'the teacher tf;aining program* *The instru- 
ments used by Widl and TTD to collect data reflect ^the specific and general 
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objectives of the programs for the preparation of teachers at the institu- 
tion, * * , ^ 

4 . An evaluation program must be continuous and longitudinal in nature. 

In order for a program of teacher evaluation to be meaff»ingful, it must 
be continuous and shoujd be longitudinal/' For effective program improvement* 
l^^i^ere needs to be a continual input from the graduates of the program. As 
was pointed out earlier* industry uses a continuous model for evaluation in 
ord^r tc^ change and improve its products^ The analogous situation holds fpr 
programs which train teachers. Programs of evaluation which are not contin- 
uous nejther allow for measurement of changed that may take place in a pro- 
gram, nor for determination of developing pr;obl%h» areas. 

The models that are in use at WKU and TTU have been in continuous opera- 
tion for over eight years. \k\ the case of the model used at WKU, data have 
been collected from ^graduates during strident teaching and their first, third 
^and fifth year in the profession. Data have been collected each year from 
the first through the fifth year for graduate^ of programs at TTU, A new 
group of graduates enter the evaluation system each year,' Mt is felt that 
the systems which are in place will continue on an indefinite basis- 

5. ' An evaluationuirogram must provide for and reflect a knowledge of the 
sTate-of-the-art in educational research and evaluation -as it relates to 
teacher educat ion 7 

The state-of-the-art of teacher evaluation is in a contindal staf^e of 
change, Sinc^ the early sixties, many resources have been allocated to the 
study of eff^ct^ive teaching, and in thp seventies a number of individuals and 
institutions developed models and plans for the effective evaluation of 
teacher^. If aft '1'nst itution is going to develop and operate an effective 
plan . for the evaluation of teachers, it must constantly be^ aware of the 
research-based changes which are bein^ made in evaluation. It should be 
pointed out that NCATE Standard 6,1 states, "The institution keeps abreast of 
emerging evaluation techniques and engages in systematic efforts to evaluate 
the quality of its graduates upon cprppletion of their program of study and 
after they enter the teaching profession** (Stantjards, 1979, 11), 

* , 
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^ WKU and TTU ^ave regularly monitored the research Jiterature related to 
teacher evaluation. The basic models that were developed nearly a decade ago 
are still being used; however, -changes have been made in tfre instrumentation 
and ot^er aspects of the - programs that reflect the results of resear||) on 
teacher evaluatiof>. These two institutions have maintained continual contact 
for the purpose of sharing 'information. Reol^itly, through the efforts of the 
Research and &evelopment Center for Teacher Education at the University of 
Texas ^d others, /a network of individuals interested iri teacher evaluation 
is Being formed (Hord &Hall, 1978), 

6, An evaluatiosij, program must ref1e(5t thorough planning prior to imple- 
m gntationT" ' ' 

The implementation of an evaluation program Hithin an institution can be 
a difficuU task* Adequate planoing prior to implismentation is essential if 
the objt^ives of evaluation, i,e., program development- and improvement, ate 
to be recognized. Following are six important points that mu$t b$ consider 
in planning and implementing a program of teacher evaluation. - ^ . 
a^ Cooperation of 'schools ^ ^ ' i ^' 

Prior to the initi atio/i of any type of study of the graduates of a 
teacher education program who are practicing teachers in the schools, it is' 
essential to obtain'the permission art<f consultation of School administrators. 
Also, it may be necessary to con'sult tf-ith and have the permission of local^ 
teacher negoti ating groups. For example, on some areas of the country extef*- 
sive fpllow-up evaluation in the^'schools ^has not been possible. because the 
local bargaifling group for the 'teachers did not make provision for such ac** 
tivities in the schools, or would nat per^mit such activ,itie$ fince they might 
infringe upon the system of evaluatioo already in use "tn the^schooJs, Ade- 
quate permission for data collection in the schaoTs is essential if a project 
of teacher e(^aluation is to be a 'success. Without such permission and a full 
understanding on the part of the school administrators ' of the , use of the 
data which is to tfe collected, the project t;anno^ succeed, 
b. Subject selection ' ^ , ^ 

"Various met^^jj^s exist for the selection of su4j6cts, taking into "consid- 
eration the constraints that were imposed in (^), abov?. The ba?ic premise 

\ 
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of subject selection is that the process m\ht be a defensible system and 
should be the same from year to year. It appears that the most practical 
system of subject selection, even though it is not the most defensible from a 
scientific'point of view, is to select teachers from a pool of those gradJ- 
ate/'wiiiKyolunteer to participate in the projeci. 
c ♦ Data soyrces 

Evaluatlion of teachers can be approached from several standpoints* Data 
collection should no«t', - however, be limited to only one source (for example, 
only the graduat.e}* Frequently, in the past, institutions have sought evalu- , 

^ative data by simpTy asking the graduate to rate the value of certain^exper- 
lences that were a part of their , teacher training program. Although this is 

*^^an >nadeq(i»ate source of data, it is an important source* Data for an ade- 
qpat^ evalijation program should be collected from a variety of sources, in- 
cluding the teachers, the teachers* supervisor and peers, their students, ^and 
probably importantly, by irKJependent Observers in the classroom. 

d . QuaTitative'and quantitative 

As an,i integral part of any evaluation effort, consideration must be 
given to ^ both qual i t^ative *and quant i tat ive data col lection . Quant i tat i ve 
'dat^ can readi]y lead to analyses allowing for the use of inferential statis- 
tics, hov/ever, in many cases qualitative ^ata can be TOre meaningful and 
useful to faciilty and adrmmstralors in program development and improvement. 

e. Points of daij collection * 

The points of collection of data in a teacher evaluation program must be 
planned prior to implementation* Various plans have been used in the longi- 

/tudinal ^studies that have been reported m the literature. The most common 
systems which are in place Include data collectron during student teaching 

^ and the fir^t, third ^nd fvfth year after entering the teathing profession, 

01" daXa tollection^durtng the first year after entering the teaching profes- 

siQd and each year thereafter for up 'to five .years^* ^ Little useful data with 

* * *t ' ' * ' 

regard to program* improvement and development can be ..collected after five 

years ^because ' of a tiumbec of factors* Ft is essential that data be collected 
Tjuring the first year af.ter- entering the profession and for -at le^st one or 
more years up tg ffv€ years. If such a -plan for data collection' ^s not ini- 
tiated at the start of the evaluation project, fnuch useful .data may.be lost* 
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f Data collection^ instrifmentation and personnel training ^ * 

^ Initially a system, fof data collection must be developed. As a part of 
this system^ instrumentation must be selected or developed which can be used ^ 
to^collect quantitative and qualitative data that will measui'e the achieve- 
ment of the objectives of the preparation program of the teacher-^. Numerous 
st.an(ftrdized instruments are in existence; however^ it my be to the advan- 
tage of the institution to develop instrumentation that will better meet Us 
particular needs. As an integral part of th"e development of the plan for in- 
strumentation and data collection, personnel who will be used to collect data 

must receive special training. For example, if a system of interact ion anal* 

-t * 

ysis IS to be employed to collect data in the classrooms of the* teachers, 
then the research assistant or data coUector^^'must recewe adequate training 
m order to collect valid and reliable data, 

^ Throughout the life of the ' evaluation project, instrumentation and 
methods of data collection must be systematic. For^ example, i^Vone type 
instrument , is used one year and tdie ' fo 1 lowing ye^r' a Second type of ^instru-^ 
ment is used, data from the two data collecfl^n periods may not be compat- 
ible^. Thus, one or njore years of usefuf data may be lost in a "longitudinal 
study. Instrumentation and a plan for data collection must be developed 
early and must stay m place for several years . Also, Personnel must receive 
adequate training each year» in order for data to be compatible from year to 
year. 

r . 
g. CofTiments about planning 

If any one of the above constraints relative to implementation of a 
teacher evaluation program are not followed, the results of the proj.ect may 
be suspect and will be of little use for program development and improvement. 
WKU and TTU ^ach have developed detai led^ plans for the implementation of 
their follow-up studies. The plans are rather detailed and space does not 
permit a lengthy description of* the projects at this point. The reader is 
referred to Teacher education program evaluation and follow-up studies: A * 
collection of current efforts (Hord & Ha 11 J 1978) for T"fnere complete de- 
scription oft the implementation of the projects or the College of Education 
^f the two institutions. 
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7. An evaluation program must i^ef lect the most efficient use of re- 
sources for data processing^ storage, retrieval and anaiysesT 

Prior to the start of 3r4eacher evaluation Project^ an institution must 
inventory its resources rel/ti ve to those avai >able for data processing , 
storage^ retrieval and analyses, A decision must be. made at fhe start rela- 
tive to the amount of d^ta to be collected and stored and how U can be re- 
trieved. Access to a high speed digital computer is essential if a large 
scale project is attempted, For> example^ one -institJtion attempted to repli- 
cate the teacher evaluation model that ^s in use WKU. ' The institution 
found tnat after collecting data for one year on a sample of 30 teachers^ the 
computer available on their camPus did not have adequate storage space avail- 
able. As a result the data analyses 'had to be earned out m a cumbersome 
manner resulting in tne need for addi t lonal ' resources, 

WKJ and TTU have been fortunate in having access to large scale compu- 
ters wTth' adequate storagjT The problems of dSta storage, analyses ano" re- 
trieval have been minimized. It should be pointed out that both institutions 
ha^e had senous problems witn data management and control, these proble^ris 
have been due to the large amount of data collected each year and kePt in 
continuous storage for yse in making comparisons across years and for special 
analyses. ^ 



An evaluation program must Provide for an effective communication 
system lor inPu!^intQ the evaluaticin Process and reedPacK ot evaluation 
TrTi 



vernation 7 

Therj^(^ust be an effective system of communication for input into the 
project from the faculty who will eventually use the data and results of the 
evaluation program. In turn there must te an effective system for corrtmuni' 
catton to the faculty such that the results can be used in pr^ram develop- 
ment and ifnprovement , The TOSt efficient system for providing faculty input 
into the Project is to make certain at the outset that the faculty feels il 
is their evaluation prograin and therefore their input is of value and use. 
It should be pointed Out that NCATE Standard 6,2 states, "The institution 
regularly evaluates its teacher education programs and. ihses the results of 
its ev.aluatior(S in the modification and improvement of those programs" 
(Standards , 1979, p. 11) . WUhout an adequate system of feedback into the 
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program for the preparation of teachers, the latter part of the Standard 
cannot be met * 

WKU and TTU have developed programs for the feedback of information into 
the programs for the preparation of teachers," Both institutions use a sys- 
tems approach for thg operation and evaluat Ion of their programs. Thus, 
there are natural built-in feedback loops* On the more practical side, the 
institutions prepare reports on a regular basis that are made available to ^ 
faculty and administrators, as well as special reports and papers relative to 
specific areas and studies, 

9, Ar) evaluation program must provide for a workable system of evaluation 
data utilization in program decision making. 

^.T^e success of an evaluation program may be measured by the use that^is 
ma'de of the results. If the results from the program are not used in a mean- 
ingful manner, then the evaluation program may ^ total failure* Dissemi- 
nation *of the results of the evaluation program may not be sufficient to make 
Changes and needed improvements in a program for the preparation of teachers. 
The institution must have a built-in systiem for the utiliz^ation of t^he re- 
suits of the evaluation program* As was noted under (B), a systertls approach 
to the training of teachers can have ^ built-in subsystem or feedback loo[j 
that will make possible the use of' the results of a teacher evaluation pro- 
gram, 

1 0 , An evaluation 'program must have a component to assess the effec- 
tiveness of the evaluation Process, 

Industry constantly evaluates the standards and evaluation procedures 
that it emf>loys in the evaluation of its products. Any evaluation program 
should have a built-in system for assessing its ef f e(;tiveness as a tool for 
gathering data and making meaningful decisions. Therefore, the program for 
self-assessment may take the form of the external review by consultants or 
an Internal review by the project staff. Such a subsystem within the evalua- 
tion system should be developed at the beginning of the project, WKU and TTU 
have developed systems for review of their evaluation models* 
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Summary and Recommendations 

This paper has described ten characteristics that must be takeh into' 
account in the development of an adequate system for the evaluation of teach' 
ers. To reiterate: (1) faculty, administration and goveri^ing body commit- 
Tnent to the evaluation process; (2) the evaluation program must be an inte- 
gral part of the total program of teacher education; (3) the evaluation pra- 
gram mu'st be based on objectives and be specific* yet -cover the -general edu- 
^tijon of teachers; (4) the program must be longitudinal and continuous; (5) 
the program must reflect the state-of-the-art of evaluation; (6) 'the program 
must reflect thorough planning; (?) adequate computer resources must be 
aval lab le; (8) there must be an ef feet i ve feedback system; (9) results of 
the evaluation must be usable in decision making;^ and (10)' the evaluation 
program must >iave a self-evaluation component* If these characteristics are 
not inherent in a program of teacher evaluatiorv* then th^ system may not be 
adequate, and consequently the results may not be usable for program develop- 
ment and improvement, ^ 

Although this paper describes evaluation of classrt?Om tearhers* grades 
K-12» the design characteristics^may be applied to almost any model for the 
evaluation of personnel completing programs oriented toward professions in 
the schools (e,g., principals, superintendents* curriculum supervisors, coun- 
selors, and school psychologists). It is recommended that any institution 
establishing a program for teacher education evaluation give ser/ious consid- 
eration to the design characteristics outlined in this paper. It is. felt 
that the institution will have fewer problems, a more efficient system that 
wi)l meet NCATE standards* and furthermore will be more useful for program 
development and improvement , ^ ' 
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FINANCE AND RESOURCE ALLOCATIONS RELATIVE Tp 
' TEACHER EDUCATION EVALUATION 
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Tennessee-Jedinological University 



^ It is sad but true that many, if not most, insti.tutions of higher' learn- 
ing have neglected the evaluation of their graduates. Teacher education in 
this instance, however, has gone far beyond most of the oth^r professions. 
Central administrations at mjst colleges and universities have been reluctant, 
to fund the progr ams for the evaluation of graduate's ^ Thi s, indeed, is 
strange in a time when more and more afiphasis is being f^laced on the "qual- 
ity^' of the instruction being provided; namely, how well the graduates per- 
form. This impetus without a doubt has been related to the expenditures for 
various programs. ^ 

In Tennessee the Tennessee Higher Education Commission has implemented a 
system whereby each institution is evaluated annually based upon several cri- 
teria that supposedly relate to the quality of instruction. One important 
criterion is how wel 1 graduates perform on specified objective measures* 
This is one of the first efforts in the nation to condijct such an evaluation. 
Irrespective of the various constraints in conducting evaluations of gradu- 
ates, the fact remains that if programs of teacher education are, to be im- 
proved there must be an ongoing evaluation of how well the graduates perform 
as they exit their programs of study and after they enter their professional 
roles. No matter how critical one may be of teacher education, it is obvious 
that the profession is trying hard to increase program effectiveness based^ 
upon results "fn>fli^ fol low-up studies and evaluation of graduates . The only 
national accrediting association for teachpr education tNCATE) is placing 
more and more emphasis on the evaluation of teacher education programs and 
graduates. In 1979 more institutions were denied accreditation because of 
deficiencies in governance and evaluation (Standards 1 and 6) than in any 
other area (NCATE, 1980)* 

The entirg issue of denying institutions NCATE accreditation if they 
failed to meet all of the standards in a gUeir family of standar'ds, namely. 



\ 



59 ^ 



ERIC 



( 



the Is, 2s, 3s, etc, which would result in automatic denial of accreditation, 
came ^out because so many institutions were failing to meet the standards on 
evaluation and governance. The first vote of the Council on this issue 
failed, by one vote. A year later the issue was again raised by the Stan- 
dards Committee and the Council voted to approve by two voCes, Since then, 
the Council has asked and approved holding in abeyance any further action un- 
til more input can be gathered and analyzed from the various constituencies 
of the Counci 1 , Stemming/- from the .actions of the Counci 1 ^related to the 
family of standards issue was the development of Evidence Questions relating 
to all of the standards ^ a modification of the standards on goverjiance. 

It goes without saying that teacher education evaluation must be con- 
sidered ^ necessary part of oyera]l teacher education programs. The impor- 
tance of teacher education evaluation is unquestioned, -This being true, ir- 
respective of the'^tKATE or other standards, there must be adequate resoui'ce 
allocation and funding for the many activities that must 'occur. Those indi- 
viduals who control resources in an institution must foresee that evaluation 
is absolutely necessary for program modification and improvement. If a pro- 
gram is to be accredited or reaccredited by^NCATE, there must be a systematic 
effort to evaluate "program and product^" in teacher education. This evalu- 
ation must be directly related t^^ the objectives of the program. 

s The cost of effective an?l meaningful teacher education evaluation often 
requires a re-examination of available resources within the control of the 
institution. It is unfortunate, but it does not seem likely, that the pros- 
pects for outside funding "for evaltjation will be forthcoming. Very few funds 
from outside sources have been al lotted to evaluation ^or.ts , In some 
insti^Ut^pns that have education foundations, some funds have been utilized 
for foUow-up studies in Vhe evaluation of graduates. In tine main, however, 
it appears that the fiance and resource allocations will have to come from 
the university, colleg/, or department. The OTOunt of finance and resource 
allocations is dependent upon the comprehensiveness of the teacher education 
evaluat ion system. Probably, the greatest cost wi 1 1 occur for the initial 
implementation of the evaluation system. After th^ system ' has t)ecome 
operable,, the maintenance of the systenj will not be so costly, During the 
first year thet-e must be considerable resources applied for staff development 
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time, for planning, training, etc. The key factors related to cost, of 
course, will be based upon the design, inst^rumentation utilized, and imple* 
mentation of' the system, The amount of money to be expended wtll vary also 
according to the size of the institution, the number of graduates, sampling 
techniques, whether direct observations of graduates are to be made, etc. 

As a medium si zed , multi-purpose university, Tennessee Technological 
University has developed an evaluatfon system based upon the AACTE model. 
Basically, the system incorporates four modes of gathering evaluative data: 
(1) students; (2) self; (3) immediate supervisor, principal, etc; and {4) 
trained observers. More information concerning this system is available from 
'the Col lege of Education, Tennessee ^ Technological University, Cooke vi 1 le, 
TN , 38501 . Fol lowing is a summary of the approximate project costs at 
Tennessee Technological University for 1979-80 (Ayers, 1976). 



Estimated Yearly Costs for Condycling Follow-up Evaluation 
Studies at Tennessee Technological University 

* ' * \. 
3, 1/2-time Gratluate Assistants fdr 9\mos. ' , .$7,155 

academic >e^r'at $2,385 eeTch " , • '' 

1/4-fTian year for Secretary at $9,400/yr. 2,350 

Mileage for visitation in classrooms 1,400 
7000 mi-les at $0.20/mile 

' - ♦ ' 

Communications (postage- afid' telephone)' ^ 600 

Supplies and Printing' {paper, envelopes, forms, etc,)' 500 '"^^''^ 

Total Direct Costs _ * $12,005 ^ 

3/4-fnan year for Professional Employee. ^ 6,000 

Indirect Costs (overhead) ' 3,000 , 

Total for all Co&ts . ^ $21,005 ' " 

Elimination of the requirement for extensive visitation could potential- 
ly reduce the Direct Costs for the project to about $5,500 and the Total for 
all Co$ts to about $ll,o5o. ♦ ' - 
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'Reasonable follow-up studies can be implemented on a limited budget and 
still provide very useful information* Follow-up mail surveys can be carried 
out at a cost about $0*50 per subject (exclusive of personnel costs). 
Such surveys are essential for gathering routine information for the develop- 
ment and implementation of larger studies such as the Tennessee Technological 
University Te^acher Evaluation Model . 

The Evaluation Model can be implemented at smaller institutions 'with* 
limited budgets. ^ As was noted earlier, data about the graduates are col- 
lected from four sources, i.e., the graduate, the supervisor or principal ^ 
studeots of the graduate, and independent observers. The largest costs are 
associated with sending independent observers to work^ in ^ the schools to 
gather data on a firsthand basis. If this later step can be eliminated, the 
costs associated with the project can greatly reduced. Jjs the present 
time studies are underway to predict thfe results ► of the data gathered by 
independent observers based on data from the other sources. 

Personal data about graduates can be gathered by mail surveys, from uni- 
versity records* and telephone cal Is , Information from employers can be 
gathered through structured, mailed questionnaires and telephone Cdlls. In- 
formation from students of the graduates can be gathered by mailing survey 
instruments to teachers and in turn asking them to administer these to their 
students. Direct evidence of classroom interaction can be collected by such 
individuals as student teaching supervisors or principals (It should be noted 
that these individuals must be carefully trained). Also, data can be col- 
lected by use of audio tapes or video taping in each teacher's classroom. 
Ag^ain, these items, can bje handled by use of the mail system. 

Altjjough the validity and reliab*ility of the data may be lower, it is 
felt that much useful research data can be gethered,. using the above tecH- 
niques, that will aid in satisfying NCATE standards*^ In turn, and most im- 
portant^ it is felt , that ^ improved systems for the training of teachers will 
result. 

One possible way .to improve teacher education evaluation systems and de- 
crease costs would be in establishing a national network of institutions^ 
seriously involved in teacher education evaluation. Through such a. network 
there could be a sharing of results of particular sources of data or effec** 

* 
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tiveness of changes fnade in programs. Al5o» assistance could be provided- in 



selecting alternatives which would ^j^ave a higher potential for Success* If 
such a network were established it would be extrediely important for all in- 
volved to be participating and not simply being involved just to garn infor- 
mation* * 

In conclusion, following is a list of considerations on how program 
evaluations may occur by taking advantage of existing resources {Adams, 
1981). ^ ^ . . 

1/ Take advantage of field-based faculty activitie$^* Supervisors of 
student teachers and inservice activities' provide, opportunities fpr faculty 
to collect follow-up evaluation data. Appointmenijs may be made with gradu- 
ates to obtain evaluative information to coincide with other business in a 
given location -or area- 

2. When students are evaluated dur I'hg the pre service program, these 
data may be designed to provide useful information about the student's pro- 
gress. Similarly^ systematic and standardized supervision of student teach- 
ers may provide valuable "exit'' deta at little or m extra cost. Entry data 
that can be us^ for evaluation purposes can be obtained through the inclu- 
sion of selecte<J instruments in the initial packet completed by prospective 
teacher education candidates. 

3* Limit the data collection activities to those students fn counties 
or school systems within a reasonable geographic location to the university, 
Thi5 may, however, affect the validity of some of the findings. Care must be 
taken in studying the data. If the validity is questioned it^may require a 



curring basis* This is particularly viable if school districts will help in 
collecting some typei of data not available through a more limited (Contact, 

5, A major expense category will probaWy be the data processing re- 
quired for a complex -evaluation system. This is in the form of both the 
,hardware (computer), software (computer programs), and the personnel required 
to maintain the data, set. This couJd amount to a sizeable cost* particularly 
for institutions that do not have the technology readily available. One ap- 




wider sampling. 




proach to reducing these costs is ^ough cost sharing, wltfere two or more 

institutions coirld share the cost of data processing, storage, and analysis^ 

The sfJecifics woulcj have to" agreecf upon, but the costs could be: greatly 

reduced if these- institutions shared the^cosls instead of eklh dioe^p^tain- 

ing f(iem Separately." , ^ ' : ^ \ * 

6^ At m^^ be adyisableyto form multi-institutional consortia. This may 

provide a way to reduce coys, -particularly fo*r some, aspects of the*^valua- 

^ i ' «» ■ 

tion System. I|ie problems wUti mjlti-institurional arrangements are obvious: 

institutional integrity, decision malting, ^ logistics, competition for -stu- 
dents, etc. The* one problem that is a-^disad^a^fcage is the possible reduction 
in faCfilty^ownership and ultimately reje^ticm of the evaluation . process- " If 
this barrier could be oJbvi ated, the multi - institutional' i approach might be 
quite attractive, particularly to*^all Institutions. . 

It is recognized' that this paper has not included very many specifics 

^ w^Jch wouldVelate to all ins*titutions reTative to finance and rescJlirce alto* 
cations for teacher education evaluation systems. Hopeful ly, however, the 
case has been 'made that * to improve teacher education progral(n$ .ancLiJrSducts, 
there must t)e " substantial ex^nditure of (^ds allotted to teacher education 
to cari^y out a system of evaluatiort th-at^will be successful* We in teacher 

■ education must continue to try to improve\^ur programs. The ti^nes call for 
"proof" .that we are truly preparing i nd i tldij^gl s who can pei^fi^rm effectively 
in the world of the ^teaching profession. This c^Mujot be^ done without a good 

\eyalMation systfefn. ' ^ 

♦ 
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\Am IS THt FUTURE OF TEAGHER EDUCATION PROGRAM EVALUATION?^ 

. Research and Development Center for Teacher^ Educatio;i 

The University of, Texas at Austin 

. * 

* Exploring the future of teacher education program evaluation is a 
difficult and risky task for many r^sons. The teacher education program 
evaluation and follow-up (T€PFU) movement has gradually deve^bped over the 
last ten years,- the principal driving forces having been the work of ten to 

^twelve dedicated individuals and the requirement of fJCATE. standards 6.1 and 
6,^, There does not appear to be a major ground swell of interest in TEPFU 

ITT * 

just around the corner. Doing TEPFU studies is hard, complex work, sometimes 
discouraging to prospective researchers, and so' not likely to attract large 
crowds. Study designs are imperfect and the data are dirty. Thus it is 
unlikely that' earth-shattering findings will emerge in the near future which 
would grab the attention of the mass media. Also, it^ 1s a fact of life in an 
Einsteinian .universe that one cannot see the future, ^ 

Regardless^f the problems, conditions and reasons why the future can't 
be predicted with accuracy, I accept the challenge to use this opportunity to 
reflect on the TEPFU movement, 1 will also commeht on study desigt^s and pos-* 
s^ible influences that I predict for th^ next few years. In pr^aring this 
paper stftne concrete recommendations manifested, which^I would ^ike to offer 
about the directions I feel the movement should take. These recommendations 
are within the realm of what we can control, I will also do some speculatitig 
about the future in areas where|We have little or no control. My recommenda- 
tions for the controllable future mkp some sense; for the uncontrollable 
future my guesses may help us t-p anticipate some of the pressures that will 
be placed on us. In neither case do I see profound breakthroi/ghs,. great pub- 



.^Jhe research describe^tt^ifl^ was conducted under contract with the 
National Institute of EducatTonTT The opinions expressed are thosjajfof the 
author and do not' necessari ly reflect^ the positioQ or policy of th^Wational 
Institute of Education. ^ No ^endorsement by -the National Institute of 
Education should be^nferred, " - ' ^ - ^ 
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lie acclaim or a large number of^rants coming to or out of TEPFU activities. 
I do believe, however, that there are some things we can do to- enhance the 
quality and quantity of TEPFU studies and the use of study findings, 

The Development of TEPFU: 1970-80 

Teacher education program evlauation and fol low-up studies came, into 
their own 1n the 1970*s. The AACTE stimulated paper by Sandefur "(^70), the'' 
CBTE movement and the federal emphasis on evaluation of ev^ything cataly>2ed 
different ways of thinking about teacher ^education program evaluation. In 
the past, program evaluation consisted of course grades, data from the place- 
ment office about the number of graduates who went into teaching and the jdc- 
casional mailed questionnaire to post-graduates to find out what they remem- 
bered'about their preservice program. In _the 1970*s the evaluation .(!|uestion$ 
became a lutJe more focused and there was increased emphasis on follow-up 
studies, and wuhin-program evaluation to a lesser extent. The two research 
questions most' frequent^yj addressed were: (1) What are graduates^of preser- 
vice teacher education (PTE) programs -like after they go intb the afield? 
And", (2) What are the immediate effects of di fferent courses and exper- 
iences? 

^With the early lead of Adams and Sandefur at Western Kentucky Univ.ersity'- 
land Ayers and Hearn at Tennessee Technological University, data began to.be^ 
collected about graduates one, three and five years after they left the pro- 
gram, The CBTE movement encouraged program evaluaticfn efforts which 'looked"^ 
at the immediate effects of certain, program components at such institutions, 
as the University of Nebraska, Univensity of Houston, Oregon . Col lege of Edur 
cation and Weber State College, And in the late 1^70^s Kevin, Ryan recruUed 
si-x doctoral students to do an in-depth ethnographic study . of first ^y^a'r , 
teachers^ In add-ition, NCATE and a few states (e.g., -Ohio) began *o push £y 
documentation of teacher educat ion program effectiveness and fol bw-Np ^jf 



graduates. . " ^ , 

There were'oth^r studies, institutions and person^ who cpuld be identi- 
fied, bur for pur^ purposes here the ones cited profl'ide an overall picture of 
how the 70*^^ developed"! One additional trend out of the 70*s that is impor- ^ - 
tant to .note is the gradual development of an invisible college or network of " ^ , 
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the actors who are concerned with the conduct of TEPFU Studies (Cooper > 
1979). The Texa5 R&D Center, has participated in this. Note that to date 
this network has served primarily as an internal support system for TEPFU re- 
searchers "and not as much for external dissemination ands.upport building. 
In summary^ during the 197^80 geriod the following occurred: 

1) TEPFU emerged a distinct areaNjf study. 

2) All studies were done in relation fb^preservice programs, 

3) ' Studies focused on within-program eff^^ and follow-up of 

graduates . 

4] The number of studies and institutions inlolved increased, 
5) TEPFU became required for PTE program accreditation. 
5). Individuals and institutions involved established a network. 
. 7) Some of the same ipeasures were used across studies. 
8) The Overall amount\of activity* number of]actors> and data base 
increased- dramatically- 

Interestingly^ in a^decade of high federal /nvolvement in all aspects of 
education, to the best of my knowledge no TEPFU activity Kad direct federal 
support , , The awakening of the need for TEPFU studies and the support for 
' 'them ha$ been done by individual institutions and dedicated individuals with 
little or no- outside' encourageqient and reward. The TEPFU'ers of the 197(i*s 
should'be applauded for" demonstrating the kind of initiative ,and commitment 
that rneans, professional teacher educaticfn, 

pne other" characteristic of TEPFU studies In the 1970's that should be 
^entionid was the "whacking" at different parts of the teacher education con- 
tinuum "without maintaining a picture of the totality. Some institutions ^d 
^indiyidual rjesearchers devoted nearlp^ all their energies to collecting data 
i^n ;long1^:udina1 stutJies of graduates. Others careful ly tesffed indi vidual 
coqrses and modules^ while , some focused on initial teacher selection, Jhe 
studies tended to. reflect an overemphasis uport particular points, alon^'the 
^ teacher education continuum and- particular components of a teacher education 
program/ without payi^hg- equal attention to the interrelationships with other^ 
parts. o'f a pVogram>"*or the overall continuum, ^here was> f^jrthermore> very 
lUtle suctess in deva/opihg effective strategies for using the results of 
TEPFU sti|<fies to^chang^ preservic'e programs. 

If the above are- a fevt of the higtilights of the 70's">-then wh^t would be 
j reason^^le ;"ecommejidat,1ons and predictior>s for the ]980's? These are the 



67 



70 



topics to be discussed in the rematinder of this paper-.. F^irst, the framework 
of the t^eaCher education continuum will be^ described, The continuum will 
"then be used to, rais!fe questions about present study designs and to .make 
recommendations^ abgytthow TEPF^U studies should be focused in the future. 
This we can control*. The fi.nal parts erf the paper will be reserved for a few 
speculations about what lies, ahead, over which we ha/e httle or no control. 
Note that mcJst of my remarks will come from the preservice perspective ^ince 
that is wh&re the activity has been. 

The Teacher Education Continuum ' . 

One of the important philosophical' pushes at this t^me is the shift away 
front thinking of teacher education as a dichotomy consisting of preservice in 
one eel] and inservice in another. Instead of viewing^teacher educatmn from 
the point of view of distinct college courses and random inservice training 

, experiences, teacher education is now being viewed as a continuum of p^rfes- 
.sion-long development. Froni the teacher's perspective, teacher education is 
a continuum of experiences from early entry, to prese^^vice teacher education 
(PTE) ^to "loduction" {i.e., early inservice) to the career-long inservice 
phase. ^ " ' , 

* Viewing teacher education along / a cont inuum is not only important for 
program development, it is also important to the design of PTE program evalu- 
ation. To illustrate th^ relationship between theteacher educatjpn continu- 
um and TEPFU consider Figure 1. Jn this figure the various" phases along the 
teacher education continuum are named. The figure also identifies key points 
for data "Col lect^ion. O's, represent ing ^'observations, " ai^e used to mark 

' these points , Note tlrat the type of program (e.g, elementary, ^secondary, 
CBTE) or. the types of measures (e.g,, questionnaire, observation, interviews) 
do not need to be differentiated here. The assumptions -are tliat the frame- 
work is Simple enough to apply to most teacher education programs and that 
multiple measures would be required at each 0. Although this framework looks 
pverly simplistic, yfit can be^ used to point out, some ^^of the futility of the 
"whacking" that was done in the 70's ancK to make my recommendations for the 
fulftre* *. ^ / . 



^ In , designing a TEPFU study or. any other study the first step must be 
deciding on what questions to ask. The research question/s sets Parameters 
on the study design and the variables to be assessed. Some of the different 
TEPFU questions that could' be asked are summarised in Figure 2. 

The variables to be assessed obviously dictate the measuremerit options, 
For example question 4, "What are graduates of our PTE program like?" would, 
require a^ different study design th^an would be required 'for the question, 
'*How long do teachers maintain the skills'that they had at the time of ?Jl 
graduation?" The first question only requires one time data collection at 
point O3, The second question requi res longi tudinal data co 1 lection at 
t^'Ties 0^, O4 arid on to 0^, with the same variables assessed each timg, 
Al) of these questions and some others could be asJ<ed, but aM do not seem 
equally reasonable. Unfortunately, many TEPFJ studies in the Past have not 
f J F ly developed their questions or the reasom ng behi nd their des ign^, as 
wi ) ) be 1 1 lustrated below, 

f 

Recommendations for Future Study Questions - 

Several, TEPFU studies in the 70's pUce^J heavy^emphasi^ on the folbw-Up 
of gracJuates, The standard design was to collect data -crf tst^ 3rd and 5th 
year teachers. ^Question the utility and validity of doing mre of these 
studies, Ayers (1980) and Adams {1979) have reported little change in find- 
ings across cohorts, and that the third year d^ta refle^tec^ the ."^tiighest level 
of functiomn^'*^ (Ayers, p, 29), I contend that it tiljiighly unlikely, that 
the effects of a preservice program can be d^stin^uisheo from other interven- 
ing variables after the first 'year of inservice. The induction phase^ 'Which 
1$ not addressed in the U,S, but is in the U,K,^ is sucj) *a powerful treatment 
I hat any residual unique preservice program Effect* would be washed out , 
Even if they were there, it would Be impossible to measure them^wjth today's 
designs and measurement technologies. ^ 

If some agency has need for data about teachers' skills as they move 
through their career, then^ that agency -should support the n^cess'ary longi- 
tudinal study. It is not directly useful -to institutions who are responsible 
for initial training. The TEPFU f6c<js for cf^ejervic^ progt^afrrs should be on^^ 
(U assessing teacher knowledge (i-e.^^ cognitive competencies) and (2) their 



teaching ^skills (i*e*, performance competencies) at the time they exit the 
program^ not long term follow-up* Some'useful program development informa- 
^tion might be obtained from following a sample of teachers during the induc- 
tion phase, but there ^s no apparent reason for the preservice institution 
to follow them into fuM inservice* Now, if a tea(:her education insti*jtion 
plan^ to J)ecome involved m teacher education for the induction phase (TEI) 
then further foUow-up data would be warranted* 

Another point , which has recently been proposed by deVoss and Hawk 
( 1980) and Bor ich (1980) , is col lec ting teacher "need" data from mservice 
teachers as a part of PTE/TEPFU a'ctivities* Now, I have always had a hard 
time understanding "tVie logic behind needs assessments* I don't see how a 
teacher can tell y6u what they need when, by definition, they don't know ^rfhat 
tney don't know* So how can they tell you what they need? Tney 'can identify 
proble-Tis and concerns t^]^ugh, * 

8ut the concerns model (Fu 1 ler » 1969) and other developmental model s 
de^ionst^-ate that teachers *^t d-fferent points in time percei ve that they have 
different kinds of problems* If the developmental models are correct, a 
teacher changes in concerns a^^/he moves along the teacher education con- 
t inugrn. Of what use to a preservice program is extensive information about 
the needs of teachers five years after graduation who are significantly TOre 
matijre in their prc(fess^^||fal development? Right now we don't even address 
fir St -year teacher concerns in preservice programs. Once again, I would 
argue against mult i -year fol low-up studies -for preservice progr ajn evalu- 
at ion. 

As Tom (1980) has pointed out, PTE institutions now receive NCATE credit 
for doing TEPFU studies. Neither the qua-lity of fhe study nor the quality of 
the PTE graduate affects program certification. I suspect that quality 
checks on graduates' wi 1 1 continued to be; pushed by state competency tests, i 
hope that these quality checks will consider both cognitive and performance 
data* Clearly, TEPFU studies need to do this* Survey questionnaires with 
co#se happiness coefficients are very weak data to use in refining programs* 
Hopefully, teacher training institutions wtll, establish their own quality con- 
troU without outride force being applied.^ 

^The movement toward "cornmon core" instruments is encouraging (Schalock, 
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19^9), The^^only way that cross program comparisons will be possible is if 
the same data are collected for each p/ogram. The resistance to agreeing on 
measures may ijidicate some resistance to having cross program comparisons 
made. 

Another step which needs ^ be taken is to collect the same data at each 
point along the continuum. Some- changes in measures should be done to incor- 
porate ne>^\(nethodolagies, but any study questions that require comparison at 
two points m tme require that the same data be collected at both points. 
Although t^is IS an obvious point* why are there so few instances of t^he sane 
variables being measured for preservice and then in follow-up studies? 

Futjre TEPFU studies Should look more closely at within-PT£ program fea- 
tjres .e.g. courses, faculty, modules* field experiences) to determine what 
effects each nas , The global assessment of teacher ^Character i st ks at O3 
doe$ rvDt nelD m program refinement' unless we also know what the specific 
effects are of different parts of the program. 

In sunmary, aiy recommendations for the near future in areas wtnch we can 
cont^rol are: , 

1) Cut follow-up studies back ^0 first year follow-up only. 

2) Focus studies. on asses$ing the Effects of basic components 
%i of t^e PTE program. *^ 

. 3) Contrnote^the move toward common core measui^e|t. 
' 4) Sive some thought to what the significance ^of the induction 
phase, % 
5) ^6e sure the study design wiVl address the prespecified study 

*questions. 
6} There should be study questions, 

7) Collect the same data at different ^points in time. 

8) Although not mentioned specifically above, limit the studies to 
what can realistically bs done* and insure that the findings are 
u$ed. 

The. Uncontrol lable Future 

Much of the future is unpredictable and much cannot be controlled. My 
guesses of the moment about TEPFU over the^next several years are as fol^ 
lows:" ' . 

J) Continued boot strapping . ,1 do not see policy makers, funding agen- 
cies or program developers becoming overly excited or even particularly in- 
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terested in TEPFU findings. I do not wish to be a prophet doom and gloom, 
but" I just do not see a pot of gold, glory, or* even much interest in the work 
of a few very dedicated individual efforts. 

2) New study sophistication . The "how to's"" have been worked out and" 
the track record of the 70*s demOf^strates that TEPFU studies are manageable. 
We also have baseline data o^^ many programs. In the 1980*s the studies 
should be rnore focused, more sophisticated ' design and' should use wasures 
with increased reliabilities and validities. 

3} ' Network bui Idmg . The network of persons and instUutions if^volved 
in TEPFU actwuies should continue to grow. The network can provide mutual 
support, and increase ^collaboration. It is important to the movement that all 
'nembers be s^jppor 1 1 ve . ^ Even when there are ideological differences, U will 
be important to adcfress thes* m positive ways. The network is too fragile 
to Support w^f-ring camps. Criticise m the form of attacks, no matt,er how 
well intentioned, wMl fracture rather than help to build a stronger net- 
work . 

4) Micro-computer data bases . With increasing use of micro-computers 
for program Tianagement and ^nitoring^ an expanded data base should become 
readily available for TEPFU studies. For example, many of the process and 
cost questions should be addressable by in^lyzing program management data. 

5) Use of TEPFU finding . Developing strategies that will facilitate 
use of TEPFU findings to TOdify prog^^ams will continue to be a, problem* We 
a^^e a data collection profession, not a data us>ng one. The problem is" much 
lar^r than TEPFU, and one that seems to defy solution. Perhaps this very 
challenge is one that TEPFu people could address as part of their research* 

6) Inservice TEPFU . Teacher education occurs across the professional 
continuum. In the 1980*s induction and inservice teacher education program 
evaluat ion and fol Ipw-up studies should emerge as important act ivi ties. 
There is little or no data on the effects of induction Md inservice programs 
in this country. In fact the emphasis of the TEPFU movement could shift 
toward these activities. One spin-off of this could be a larger number of 
individuals arid institutions *vho are inte*rested#in TEPFU activities and this 
cou-ld lead to more national attention and better funding. 



The preceding have been some thoughts about the future of TEPFU, There 

are many teacher educators who see the importance of doing teacher education 

program evaluation and follow-up studies. There are even some who will use 

the findings. The movement was born in the l970's; now all we have to do is 

help It grow and contribute to the improvement of teacher education at all 

points along the continuum. 
* 
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Figure 1; The Teacher Educat>^ ContiJiuum 
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Data ■ ^ Questions ^ 

Sis What are the charactfefi sties of oreservice 

X , teacher education applicants? 

0^ Wfiat are the contents and processes' used in 

■ ^ PTE? 

Oo ' What are^^e Immediate effects of oarticular 
PTE pnJcesses and contents? ' 

O3 What are the characteristics of PTE graduates? 

0^. 0, What are the immediate effects of the PTE 

program? ' 

0^ 0^ What is the relationship between characteristics 

of graduates andtheir entry characteristics? 

^3 " ^1 How do the graduates of one orogram compare 

Ojj- O^-? with the graduates of another orogram? 

Oc What are the characteristics of teacher^ at the 

time of their full certification? 

- 0^ How does |he induction Dhase affect teacher 

characteristics? ^ 

Oj^ - Oc How do teacher characteristics change across 

years of inservice? 

Og - What are the effects of a oarticular inservice? 



75 

so 




■ TEACH£R EDUCATION PROGRAM EVALUATION ^ 
A SEAftl^H/OR ALTERNATIVES * ' 

Tom Savage^^ * , * ■ 

^ Texas A&M University 

Any discussion of teacher education* program evaluation .and - fol low-up 
is certain to raise the anx-iety level of ^e4cher educators. The discovery 
that weakness \u program ei^laaUon i.s a major cause of acCreditation,.xlefi>a 
by NCATE is cert-ain to elevate blood piessure of. those charged with program 
evtaluation responsibilities. What causes the anxiety is a lack of knowledge 
of alternative evaluation, models. This void is clearly illustrated by- the 
emphasis that has been placed on the practice of mailing questionnaires to 
graduates and their principals. When the obvioiis weaknesses of this data 
gathering practice are pointed out to teacher educators, they .have diffi|:ulty 
stating alternatives which are feasible' and cost effective. 

Perhaps the first step that is needed in generating alternative models 
ts to address sorfte basic questions- aod assumptions which . guide curj^-^nt^prac- 
tice. This^ is an especially crucial task because many teacher educators feel 
that accrediting agencies are operating on some\ssumptions that have ques- 
tionable valiciityt A case in^point is the assumption that^data suppTTeb by 
graduates and principals Qf, graduates is a sufficient cjata base for making 
program decisions. 

The typical expectation that a follow-up of- graduates is needed to 
determine program effectiveness appears to have a certain ^unt of face 
validity. However, a bit of probing uncovers some" conceptual weaknesses*^ 
This requirement presumes that there is a great^ deal 'of similarity in teach* 
ing environments and a consensus Tlf criteria used to^valuate teacfrer effec- 
tiveness. Such is simply not the case. The criteria used by one principal 
might be quite dif^ferent than the criteria used by another principal. This 
expectation seens to assume that all da^ta are_ of equal worth and only one in- 
terpretation of ttie data is thought to be feasible, regardless' of the crltffi 
ria used by the person performing the evaluat.ion or the environment where the 



teacher 'is employed. If a negative evaluation is received it is interpreted* 
to mean that somettiing is wrong wiy) the preparation of the individual* 

Proponents of the follow-up studnBSuse the industrial model as" an exam- 
pl^* Jf*ey point out that just as industj^ is willing to gtlaFantee *a pro- 
due^, so s!hoJLi*jd teacher education Institutions be' evaluated by the perfor-^ 
man c^ of their products. It should be noted,* " however » 'that most mam^fac- 
turers gualgLOtee a product only if it is used as intended.-* /\ ca^e migtit be 
made^^Jiny teacher education graduates are placed in settings "whefe they 
are unable to perform as intended.' ' ^ \ 

■ Another unstated assumption of follQw-up studies is that the teacher 
^preparation program has^some long term effect on a givei> graduate. There are* 
gj^nds for challenging this assumption, .Numerous irjiervening variables such 
/^s tbfe^ nature of the seating, ' pressure from^ teach'ing jD^^rs» ttie style of the 
admin istrato^r and^ tfte 'availability of inser^ce opportunities either rein- 
force or negate the^f fe'cts^ of a- teacher preparatiqn progratii** To expect a 
"ydung Inexperienced teacher to persist in certain beh^avior pat^eVns^ acquired 
in their, preparation program ^at might b^rt odds "with the stalus quo Is Un-» 
' real isticy^ - \ ^ ^^^^^ 

While most teacher educator^s dg accept the notion 0/ being held account- 
* abie for Ihe quality of their graduates, they wonder^ ^ they ^should be,he1^ 
accountable for the performance of'^graduate^Mn alV types of sjtuations and 
for an indefinite, period of time fol lowi ng graduatipn, JJhile there are many, 
call?* for, ''proof'*^ thit t^ach^ education - "instittjtions ?re ^.preparing 
. individuals who e^n perforof' well as teachers, >rtiat sort of d^ta is*needVd to 
,supplV that proof? This^ is one of the major issues which ne^ds to t)e 
addressed* : / , . ^ , , ^ . * , ■ 

The Hal] paper "in this Vi'bli cation assists in the sCear^for aUernatfve 
modeU by suggesting that' teacher educatiort be viewed 'as., a continuurj^ whi^ch 
includes^several evaluation points, ^actt ,q[f 1;)iese program evaluation points 
poses a different ,questijjn .and^ requires differejit data* It might be added 
th.at different, agencies should be responsible for data gathering at these 
' di/ferent^ev^aluatipn points/ J ' . ' " \^ ' 

For £xampl6/the pro^^^rtf evaluation conducted by institutions of higher 
educaticyrt should be fooMS^d on the pr^dgram oljjectives and the* accumulation of 




evidence relating to, whether or not a "Candidate has we^ those ^j^J^tves. 
This^type of "program evaluation "model would have the advantage- of providing 
feedback that could be 'tied 'to specific program components. 1 This woul'd 
facilitate program improvement an(f c-hange. A rather: rigorous set of e^it 
level requiremeats could "include such things as the^ impact of the student 
€fchers on learners* ^iichievement. ThiS' woyld require greater emphasis aod 



attentio^,.,itL,.tfife student teaching component." But' this is something the 
instituTTon can and &h"ould control. Any disct/sslon of program evaluation , by 
accreditart: ion agencies and other interested parties could then be "directed ta 
the appropriateness of the /program objectives and' the qua^lity of evidenc;^ 
osed/Co support program an^^'oertificAtion decisions. 

r The follow-up of graduates might best be assumed by the .stke education 
ageWy. The model beting developed Jn Georgia which includes re*gionah"assess- 
ment centers staffed by individuals specifically ^trained to do in-cla^ ob- 
serviatfon and charged with the responsibility of evaluating all fir^tyyear 
teachers fits we.ll )^ith this concept. Data could then bp gathered* regarding 
the performance of graduate^ *a., statewide Ij^sis using a consistent set of 
criteria. Th.is_wou,ld be a m^ sounder evaluation pro.cess than what is^cur- 
rently practiced, ^ * 

*^ conclusipn»*what ij r^efeded ^ or^er to advance the/state of " the art 
are arlternative program evaluation models. The absence of viable models has 
led t05,high levels of .anxiety and frustration. A <;riticaJ element in gen- 

Wher^ 

of aa effective evaluation program, ■ ' 

>■ , Participants 

Raymond Archer * ■ ' Roj^tjiiey H. Johnson 

Northeastern Oklahoma State Uni-versity University of >ii5ccmsin - 

' ^ , ■ " Eau Clair 

Caseel 0, Burke ^ ^ Bill Lampkin 

Weber State College ^ " ^ ■ * .Baylor University 



^ati^ these alternatives .is th^^ questi"(miTig of some/ basic assumptions, ^ 
hert this' is done* aU|p^Rate^3p^1^can be prpp^ed which meet the criteria 
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PROGRAM EVALUATION: -WhAT, WHEN, AND iiOW? 



V' ' Ted White ' 

^Glassboro State College 

' ' - ^ * n . ' 

There appear to- be four general areas in wWch .discussion was focused 
bj? the participants iri oar group* The areas break out something lilce this: 
\) What. should we be evaluating?;" 2) When should we evaluate i-t?; 3) How 
should we evaluate it?;tand 4) What should we do with it? Each question is 
discussed separately, ^ * ^ - . 

that should we be evaluating? 

In the case of this first question, there was consensus that ttie overall ^ 
effort at program Aa\uation should determioe the extent of match *etw^en 
program outcomes and professional conjmunUy needs, That.is^an effoVt needs 
to be made to shape, '/n evaluation system that will yield data useful in as- 
sessing the extent to which this match exists or doesn^t exist, pf course, 
the p<Jsition presumes th'atT on tRe one Kan^f^e have a liandTe on' prolfessional 
community rieeds and on the other hand we can devise an $*altrafivfr system ca^. 
pablfe of assessing the degree of program/corranunity match* The^ second part of . 
the ffrst question fiAhat) involved the relationship between the p>;ocess eval- 
uation and product evaluatiion" of the program. Follow-up by itself does not , 
represent total program evaluation in the same my that front-end evaluation 
by Itself jdoes not constitute t Q t a \ ppoyam^ evaluation. There is a need to 
articulate internal instructional evaluation {courses, ' adm^ssion^,^in^ruc-' 
tioji, clinVcal work) a'tong the way with the* evaluation mebhods to bfi employed 
during fol lo*y-up . ^ 
' 

When shoulrf we evaluate? " ^ , ^ . 

I- have to art extent toucRed upon the ifecoa^ di;ea of discussion, namely , 
when should we evaluate pV^ograflti. Again, there was a general consensus that 
fol low-up * evaluation, by itself will not speak to' the multiple concernsYtJfc 
program, evaluation* A greater effort needs to be undertaken in longitud^R^ 
ita c&'llfiction with prescribed access points iFor data, collection identified* 
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The opportunity wiO]*then be available to q^se t'he data formativ^ly with a bit 
more 'p^recision ^han were we to rely solely Dn fol low-up data exclu- 
'sively, . . ^ ' " * ^ 



bw should we evaluate? 



The third general question aci,d the one which generated the most discus- , 
sion was: How shc^ld we evaluate what we're evalu^ting?^ There appeared to* 
be widespread agreement' that an earl/'-identif ication of instructional - and 
program objectives wouVd be a necessary precondi'tion for the design ^nd irf^- 
plementation of comprehensive e^jaluatioa, ' The full involv^fnent of the, teach- 
er education Community (perhaps includ'ing the practicing prjbfessiofl) would be 
essential ^or bui Icling ownership and giving the evaluation proftes^ validity. 
Third, overt teacher behavior during direct classroom observation is insuf- 
ficient^by Itself- to be considered program evaluation, Wtiat and how teachjers 
think may be as important as 'how they act. Also, long-term ^fol low-up (beyon^^ 
one or two years) may not be very reliable ar\(\ therefore probably not cost 
effective. That may be true to the extint the traimhg program is cpnceived 
as a ,four or five year inclusive period with no systematic continuing in- 
volvement of the program beyond the^fourth or fifth year* Should the system 
of training be viewed differently; that i$ occurring beyond a fixed four or 
five year period, the evaluation, strategy wi*ll ' li/.ewise need to ^ be vlewe^d 
differently. ^ ^ * ^ 

What should we do /with it? 

The last question dealt with in this' group was. the "sa what*' ({uestfon: 
*What shouli/ be do With it once we've done *1t? Again, there was general 
agreement on two poi/its* First, there 'needs to b§ a^elilJerate 'effo<'l to 
' create the feedback loop between* ev^aluation and program , development (or re- 
development as the case may 'be).- If'^e complete .the/evaluation only to have 
the data -spend Us days in archives we obviously are not serving' the- profes- 
sion's best interests, ' ^ ' 

JLas'tJy and of substantial importance wa^ the universally held^ belief 



that TERFU should provide the vehicle for the presentation of evaluation^pro-^ 
grams currently in' place or under development with a discOssion of conceptual 
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features^ operational preconditions , implementation problems and benefits , 
■ cost effectiveness and most , importantly what difference has been made as a 
consequence of 4he program, The s'Bnse o1**sharing that was evident in this 
entire program nieeds to be /^panded ^l^^^-^^^^twork of people ^ready to share 
i,xieas, success and failure, (^Fbr us tt> do less would be to ignore the evalua- 
^t^ion evidence" thqt the session^pcQ^ticed, ^, * * . ■ 
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> .PROGRESS irT EVALUATION OF TEACHER EOUCATI'ON^^ 

^ Vernon Dravland " " 

* UniversUy of Let'hbrldge- / ' - - \ 

Ev'aludtors and researchers in educatltJn operate" wttftin a great di lem- 
ma. Aspects of this predicament were evident at this conference. To illus- 
trate^ let us draw an analogy between education and' one of the sciences,, for 
example, mathematics, Jhe knowledge bases of education and mathematics are, 
perhap^^ too diverse for comparison. However, we^n malce a rather i/iterest-, 
ing compai'ison if we compare the levels of ignorance in the two areas. The' 
ignorance i<i education is very unsophisticated^ while that in mathematics is 
very sophisticated. It 'takes another mathematician recognize the igno- 
rance displayed by a mathematician. However, even the *^most igrtorant" can 
propose what may appear to be very good solutions to very confjjl icated prob- 
lems in education. The key ward her^ is "appear" since research in education 
is? actually much more complex than that ir> mathematics, partly due to the 
larg? numbers of extraneous variables which are difficult to control. To use 
another example, during the development of the atomic bomb in the 1940'^ 
scientists were able to d^/ine ar-eas of "'ignorance that^had to be overcome in 
order to develop the device. Tearns of researchers were then assigned to each 
problem area and one by one the areas of ignorance were eliminated. Defining 
areas of ignorance in science is not always possible, but more likely in 
^science th^n ,education\ 

He dp not know what we do not know in education, and therefore cannot as 
easily define thp variables ^»r ar^as that requ^ire study before ve. can* solve a 
particular problem. Research in both education and science tends to create 
more questions than ^ answers. However, In education these questions 
tend to diverge from the initial prpt^lem. In Vie ph^cat sciences if an 
a^nswer is not found, the research tends to at least^elp clarify .the que s- 
tioa* This does not^jmply tfiat tfiere are n^ surprises i^ t^e results of re- 
search jn the sciences, but the questions that ' arf^^ arebmore likely to be 
convergent in nature. The situation in evaluation an outgrowth qU' the 
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Situation in education research/* Since our knowledge base\ which normally 
comes from research* is inadequate* th^ theory of evaltWition is inadequate, 

it became obvious from the paper presentations and discussions at this 
conference that researchers and * evaluators in education are concerned not 
only with the knowledge b^se, but also with instrumentation, procedures and 
administration of projects. Again, in comparison wi'th the instruments ^avai 1- 
able for physical' scientists* those available for edueatio;i re^earch.ers are 
highly uns^histicated. Researchers in the sciences are continually develop- 
ing ^new procedures. These procedures and Uchniques have always ^ been much 
more precise and universally accepted than those iT> research or evaluation in 
education. In the sciences e]iminating areas of ignorance is like fitting 
pieces into a jigsaw puzzle. In education research and evaluation we ^seem to 
"y^ack a well-articulated "framework" within which we can define our tasks and 
* our areas Of ignorance. This may be the reason for lack of a unified thrust' 
m education. 

During this conference the problem of evaluation of teacher 'education 

J 

progr^ams_ was attacked from maoy and various points of' view. Discussions^- 
focused *n everything from the initial conception of the evaluation design 
through'the admini^trition of the project and the resulting changes in the 
teacher education jjrogr'aip^'^ack' to the conceptibn of a new design. In-the 
discussion group it became^quite obvious that important philasophical differ- 
j^nces "among the various insti tut ions might necessi tate that di f ferent ap- 
proaches to the evaluation of teacher education' programs developed^. 
Oespite^e desire expressed by participants for^adequate evaluation of all 
aspects'.df tfife teacher education program, the primary concern centered on a 
system for \ generating ctiange^ in ,the program' based on the results of the 
evaluation. Therefore the central problem appears to be the establishment 
within each teacher^^educ^ion institution of an organization whio^ would 
design ' an ^^j/aj^nltton system that ^ould (1 ) ^e acceptable to th? peopfe 
responsible for the teacher education progr^am, and^(2) develop a systetn with 
the most V^ffectiVe "st^te-of-the-art'* procedures and technrttjues* in order to 
-ma^intain ctedVbility within^the institution^ as well as in the teacher educa- 
tion academic cojnmunity. The presenters made mny excellent recommendations 



that could help solve probleftis confronting those invoUed with evaluation of 
teacher education programs. 

.Education research and evalO^ation are very young comijared with research 
in the sciences. However, the last ten yeS^s have ^een considerable prog- 
ress^ We have a ntore adequate knowledge base and hence a more adequate 
theory of evaluation. . - ' - • 
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GROPING,.. AND COPING, .\ 
. WITH 

CONTRADITTORY PROGRAM EVALUATION REQUIREMENTS 

Gary G. deVoss 
The Ohio State UnieersUy 

Lyn Gubser's opening remarks made most impact on our. study, group 
of any speaker's. His remarks made such an impact, as a matter of fact» that 
they cau^sed a certain amount of. restrained panic among the study group. Sev- 
eral issues were raised concerning theory: "How should follow-up attempt to 
separate program effects from individual or extra-program effects?" or "Wfrfet 
are the hidden complexities within program and teacher-effectiveness evalj^- 
tion?" But far ami away the tnost frequently-repeated question was, '^How can 
Lyn Gujbser tell us not to use questionnaires when we only have money for 
questionnaires?" After processing si mi lar questions a bit* the 'uKder lying 
quest,i6n that emerged asked what^V'ays acceptable alternative models of fol- 
low-up could be disseminated and implemerited . and be given posi tive *approv^l 
by a visUing site evaluation team from NCATE. This Study group was saying 
if wanted to get away from questionnaires and try new avenues, new tech- 

j^niques,. new approaches, but was scared to death to do anything which might be 
perceived' as unsound or unsafe. ■ Ergo the profusion of one-shot question- 
naires Adams spoke of--not only does the questionaire provide safety in num- 
bers ("Everyone else is doing it**), but it shows ^ that something is being 
done', evert when no one iry the inslitution really knows what to do or really 
knows what NCATE might be looking for as ^he benchmark of a satisfactory pro- 
gram* ^ • * ■ ^ 

Without ^ftbubt'the inost significant points made in the ensuwg discussion 
were that a) with some help. Institutions could design creative, meaningful 
follow-^up studies quite easily, b} institutions were poised and ready to 

.implement these innovdt^ive strategies as# soon as NCATE made it cle^i^ that 
they could be implemented w1tjg)ut the ultimate penalty--nor^creditation, and 
c) a set of general .gu^delines^would be vastly appreciated as schools experi- 

,mented with different ways Uo collect data, flie first point was cogently 
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demonstrated as the group spun off myri ad , ideas for doing ' fol low-up e'valua-^ 
tions. These include; modifying the questionnaire technique by reducing the 
number sent to 10-253! of the population (randomly drawn), then attempting to 
% -contact and obtain a return from 90-100%'of the sample; interviewing by tele* 
phone to reduce transportation costs; asking volunteer groups of teachers to 
come to the university and sit in panel with faculty; starting\fie documenta- 
. , tion process early in the undergraduate career; enlisting students, teachers, 
-and staff personnel (such as university supervisors) in th$- task of keeping 
voluntary '^critical inci^dent" logs; using' extremely small samples of teach- 
ers, but keeping extremely "thick" records- of their experiences, concerns or 
problems; usin^ traditional ethnographic techniques either at the same time, 
or in place of, survey-type techniques, , 

These suggestions, some of which are already being tried in institutions 
around the country, emerged in 20 minutes of Siscussion; it wastr^ly excit- 
ing to see how creative Institutional representatives could be when the sky 
was the limit, not NCATE .standards , IT 

The second point needs little reiteration. It may suffice to repeat 
th^t "experimental" projects are being piloted in various institutions at 
this time, and that institutions represented in this study group were ready 
to give anything a try so long as they would not be Reprimanded sometime in 
the future for'their Ingenuity now. 

The third point contains a clear request for facilitation, such as Hall 
^ called for whenr he spoke of networking* Institutions are not afraid of rein- 
venting the wheel, but do not like to do so because first, it makes them look 
,bad when visit^d,^ and two, it is cost-ineffective* Two activities were 
called for here* One was the prerequisite that NCATE begin to actively en- 
courage experimentation and perhaps even waive, tlie letter of program evalua- 
tion law for a period of time if the institution can show that it is pursuing 
^ome sort of program evaluation practice. The second activity would be. to 
di^^eminat^ a Han<lbook of Possible Program Evaluation Strategies , which would 
contain several variations on 't^ie H^ettern Kentucky model, and certainly sev- 
eral alternatives to the one-shQt questijonnair^e model/ This handboiDlt, per- 
haps coupled with a toll-free "program evaluatTDn hotline" number, woul0 ap- 
pear to be the equi^valent of an.on-sifc^ expert. The group felt such a hand- 



89 



book could be written today; thgy also voiced the opinion that they would 
certainly try anything if a guarantee against censure for daring^ to do so 
would be made by NCATE. ^ ^ - ' 

As one partifcipant remarked* "It's time we started jneeting institutional 
needs rather than NCATE needs. The one-shot questionnaire ain't great and iT 
doesn't tell us anything* ,but- it meets NCATE's needs, So we're goirtg to keep 
sending 'em out until we hear different../' Maybe this session was the start 
of something different. Let's hope so. 
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FACULTY REACTIONS TO PROGRAM EVALUATION: 
• A LITANY OF "YES, 8UT*S,,/* 



Len Froyen - ^ 

University of Northern Iowa 



Evaluation, although criticail to effective program planning and de- 
velopment', does* not receive nearly the attention it merits. The two most 
comnnSnly cited reasons for this situation are 1) the cost and logistics as- 
sociated with conducting evaluation, and 2) the difficulty of converting the 
results into program improvements. Faculty resistance, however, ^s another 
factor which helps to keep evaluation in a subordinate position. 

Faculty prize autonomy,, and any effort to improve accountabi 1 ity is 
viewed as an irttrusion on academic freedom, ' The keystone of effective evalu- 
at ion --precisely specified and agreed-upon objectives --introduces expecta- 
tions and prescribes procedures that run counter to the preferred academic 
1 if^ of most professors. It is therefore not surprising that department 
faculties ^ive only cursory attention to consensus building when they estab- 
lish program objectives. Even less consideration is given^ to ways of check- 
ing compliance with objectives. As long as program evaluation is not taken 
seriously, these pro forma sessions satisfy the trappings of academic life. 
However, when evaluation, becomes an instrument for exairvlning practice with an 
expressed purpose to improve upon it, the agile minds of faculty go to work* 
The litany of "yes, but*s" goes something likp this. Evaluation is un- 
doubtedly a good and desirable thing, but,.. 

}) You cannot really measure the truly significant outcomes of instj^c- 

tion; evaluation tools are 'not sensitive to the thihgs that really 

count. 

2) ^The costs of.t^ime and .money to conduct a sound and defensible pro- 
;grani are not warranted by the limited use one can make of thd results. 

3) We already know what needs improving; the results only confirm our 
worst suspicions, and the changes deemed important are those ihat are 

^ doomed for political and economic reasons* 

* "4) It is iropossible to secure the cooperation of those who are to be 
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evaluated, ancj when they do cooperate, they elevate their efforts to 
produce the hoped-for results. 

5) .We cannot possibly separate the effects of training from those that 
are due to a host of other factors. ' 
^ 6) Evaluation so often becomes an end in itself, serving those who 
administer the program rather than those for whom the data are supposed- 
ly collected. 

7) The most effective and verifiable methods, performance-based apprai- 
sals, are used on small samples, and this makes the results highly ques- 
t ionable . 

; 8) The sheer complexity of the criterion variables precludes valid and 
reliable measurement. 
These are compelling arguments, ^valuators are hard-pressed to answer these 
Criticisms. 

Faculty have created a line of ctefense that can only be penetrated by 
more sophisticated studies or by external pressures which are persistent and 
unequi vocably focused. The state of the ' art wi 1 1 make it difficult _to 
counter the foregoing criticisms for some years to come. Yet facuV^y must 
be made part of the solution. They may only join the effort when legislative 
mandates, state departments of education, professional education associ a- 
tions, and accrediting .agencies demand that faculty participate in projects 
designed to identify and verify the products of instruction. # 

In the absence of such pressures to break down faculty resistance, eval- 
uators might devote more time to using already available data and assisting 
those who design the tools and administer the mechanisms fpr collecting these 
data. For example, several states now use the National Teacher Examination 
or a state licensing examination. The objectives that comprise the test 
blueprint may be used to establish baseline requirements on the local campus. 
Predictable dissatisfaction with these objectives can become the first step 
in bringing faculty into responsible participation in program evaluation. 
Reports from local school systems, based upon their faculty evaluation pro- 
grams, can also serve a similar]^ useful purpose. Performance samples taken 
from the evaluation instruments^of several school systems can serve as an 
initial method of assessing job-related objectives in the teacher education 



curriculum* 

* ■■ * _ 

The questionnaire has long been the standard fare of the prograrn evalua- 

tor. tluch remains to be lear'ned about tTie effective use of this s\jrvey in- 
strument* Tim#, cost, and the simple logistics of collecting data will con- 
tinue to make this approach an appealing one. Refinements in the areas of 
instrumentation, response rates, reporting procedures, and transformation of 
results into corrective measures should be encouraged and supported through 
collaboration among program evaluatlors* We may be well -advised to concen- 
trate our energies upon these activ|s*c<es while seeking ways to/create a set- 
ting that is more receptive to other data^collection and analysis procedures* 
The- time saved by relying on wor^ conver^tional and tried techniques may" be 
m5^t profitably spent in helping faculty* to understand and accept the posi- 
tive contributions of evaluation* 
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THE USE OF PROGRAM EVALUATION RESULTS: PROBLEMS AND SUGGESTIONS 



ERIC 



Ann Hungerfiian 
/ University of Michigan 

* 

The topic of our group discussion was identified inmedi ately and with 
easy concensus as the USE of evaluation results. This topic had been nien-* 
tioned in various ways in five of the si?t papers presented in the two hours 
preceding the discussion. Selection of this topic also revealed that every- 
one in the group was involved in pr^)gram evaliJation and . accepted the fact 
that the use of_ results is a political neces^ty as welj as a requirement for 
effective program development. / 

Our'group first, identified several immediate problems for putting evalu- 
ation results tc use> and tfi^n addressed these problems \n terms of three 
questions: To what different uses can or should program evaluation data be 
put; hQW can we incorporate program evaluation results into program compo- 
nents; and> how can we guarantee that results will be used? These problems 
and quest-ions are discu^ed below, . ^ \^ ' 

Thece v^re several ^oblems whtch v^ere recognized by the group to be 
significant. First .^f al 1 > ^different uses require different instruments > 
procedure^ or desigri^ not all of i^hich are always compatible. For example^ 
longitudinal comparisons require simi lar instruments^ whereas indicated 
course revisions^ay suggest changed instruments* Another problem noted with 
longitudinal sttidies-c-is that the validity of testing the effect of the pre* 
service p^ografh after trte first year out in the field is lessened by the ef- 
feet of graduate study. Furthermore^ those experienced in some form of pro- 
gram evaluation report often being caught in a tradition of very informal use 
of results/ making it difficult to switch to^ fiore , systematic usage>. while 
thoie jus^ b<fcginni,ng program evaluation need ibodels and guidelines to. assist 
them, Artd> not, to be Ignored is the danger that, during evaluation^ w6 can 
identify weaknesses much more confidently than we can reach consensus, on a 
Single definition of a "good'* teacher^ which sometimes leads to a tendency to 
panic at evaluation results > , rushing precipitously into course or program, 
change. 
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As these problems were identified, it became clear that /there are. no 
easy .solutions, but the group did look, at questioas which mi^ht cover more 
general ly the complications involved iri results usage* The first most likely 
question to ask" is, to what different uses can W should ,'program evaluation 
data be put? Our group agreed that >t can be gsed, in the foTlowing^specific^ 
ways; As an immediate feedback system to^ individuAT .stiJdents'^n conferences 
for self^ and professional development'; for curricular improvement ■ by -imme.- 
diate course revision; for long-ra|ige planning, or; radically re-shapi.ng". the 
program; for the political advantdge*of provinlg the prqgra^i^effect iye ; and, 
for improvement of the Teacher Educatijon " profession by dissemination of 
standards, designs and procedures, ' - ^ ^<^'' " ^ , 

Knowing how the results can bemused,, the next Ici'g^cal Question i,s,-^how 
can we incorporate program evaluation results into-'program components? Ttiere" 
were several viable ways pointjfe^ out," including relating evalu^tio'rt closely 
to course objectives and prografn goals. Furthermore^ the e)bjeotive-r'erated 
^v^ltiation results should be reported promptly ta involved faculty^ and 
assistance should be given to those faculty "In interpreting, resglts /- We all. 
■agreed that the faculty, should ialso be involved in the .complete cycle^of pro- 
gram development-implementatiort-evaluation-rfevisioh. . ' ; 

The third question is how; to guarantee *fhat results vrilV be used* 0^/ 
group decided that use of results must be plarined for, mafcirtg ^t a recogriized 
and- accepted program goal, and by cooperatively developing and formalizing 
procedures to attain that go^lj* . , ' " ' 

^ Jhose present identified as a critical prerequisite for the success of 
program evaluation a visible and acknowledged leader who -takes responsibility 
for the full cycle of program evaluation activities.* They- also revealed a 
high leve'it of interest in sharing knowledge, experience, instruments and pro- 
cedures relating to program fevaluatioo* There -was an. expressed^ desiVe for ^ 
some assistance' in determiairyg what constitutes ^"ideal" use of program evalu- 
ation results. Perhaps some guidelines for "use" procedures could be ^ner- 
ated and' disseminated through TEPFU? Time precluded any further or more 
specific suggestions for the TEPFU group* * * 
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' PROGRAM EVALUATION--A SYSTEMS APPROACH , 

' / . . ■ ^ James J, ^Garland 

Southwest Texas, State University 

' ' ' 

Oetroit, the Renaissance Center* and the Detroit Plaza were all -grand, 
bUt they did not^ oyer-sli^dow the .period from 3:15 to 6;15 p,m, on February 18 
in the Mackinac Bal Iroom. This ■sess'Tbn, whicTi dealt with teacher 'education 
program evaluation, ^was very rewarding, , " 

' After ' hearing six very timely and informative papers, our ■ discussion 
group convened for art ^exchange of ide^s related to the presentations. We 
first took^i&sue with the premise that tlie questionnaire technique of gather- 
ing, \fnformatiofi for program fiDllow-up y^as not a good approa£h. However-, we 
agreed that a systems --approach was required for effective program study. 
Such a system WQuld, fenploj^sev^ral techiiiqjue^s (including the questionnaire) 
for both shprt -and longer periods lof study. It was ^Iso agreed that personal 
interviews,- pOtjlip schooKevdluation teams. Teacher Centers, and students in 
the program serve ^es very effective components^ of the evaluation system, 
, " Those, in our group emphasized, as did Ihe speaker, that evaluation of 
graduates fs not,^^by Itself, enough to support a strong evaluative process. 
The system should include the admissions, office, departments, admnistrative 
offices, placement office, and oth^ir key components to provide a network of 
support agencies s_o that, each making, its own contribution, i multi-approach 
system can be implemented. , ' - ^ ' ■ 

The jgroup agreed with the presenters when they pointed out that a suffi- 
cient funding t)ase heeds to W established for the evaluation system and that 
thiV' base must be ref lectelJt in the regular budget process. Also, we sup- 
ported the id^ that a director or othelr individual shogld be designated with 
, specific responsibliti^s of carrying out the policies and procedures for pro- 
gjr^m ev^aluation and guiding, and, directing the activities. 

The grdup supported the statement by Dr, GubsV that ,?valuation 
should not be ari aftferthoiight; it.sh<^uld be \^x\t into the program*" We 
■feeT, as did tlie speaker, that evaluation must be "planned as! a part of the 
total* teajfier ■ edlicaf|foo pr<5^'t;am--not something added as an .apparent need 
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arises* The data gathered fn the evaluation process serve as a guide for the 
program* as a sounding instrument for new and innovative ideas, as a measure 
of progress toward stated goals, and as a foundation for program accounta- 
bility. /Therefore, U is essential that the evaluation model be initiated as 
an inteoral part of the teacher education process. 

As/ our group di scussion ^progressed, we mentioned several ideas and com- 
ponent/p^rts that would form a well planned and workable evaluation systerrr. 
Due to the lack of time, however, our group did not formalize a workable 
modeK On the basis of the six papers and our discussion of them, it would 
seem/that Figure 1 /^epresents * a workable system for the^ agencies and their 
i-nte/^relationships in the evaluation process* 
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... SMALL COLLEGE NEED5 IN FOLLOW-UP EVALUATION STUDIES 

■ Al Kilgore 

' * University of Nebraska-Lincoln 

./^The follow-up small group session, facilitated, by Al Kilgore, con- 
sisted of teacher education personnel from relatively smal 1 , private col- 
leges. The primary focus, after introductions, Centered on what each of the 
representative colleges were currently doing regarding follow-up studies,* and 
.more specifically, to deifribe the iparticular role each participant played in 
the evaluation process. 

• The level of involvement of the members of this group in fol low-up 
studies of their graduates has been minimal, although all programs had both a 
semi*fornial as well as informal neiwork working for them. Informal and ran- 
.dom visits to graduates who w^^^e teaching along with some observations of 
graduates in their classrooms, to questionnaires to first *year teachers and' 
T^ntenHewing gi^aduates who had returned to tampus for graduate classes wfere 
among ^he data gathering processes used. The data collected did not appear 
to be used systematical Ix for .program- type decision making, rather the indi- 
viduals collecting the data may have made- some changes in the i part of the 
progf"ams\,that they had direct control over. For example, based on informal, 
talk^ with graduates, one professor modified a methods course. At any rate, 
the final consensus was that each program had a follow-up evaluation, process 
of some type and the data collected were he>pfuK Evaluation design, how- 
ever, was very loose, unsystematic, and underfinanced* * ^ 

This led to the issue that most concerned the- participants in'this 
group; to explore and learn evaluation designs that could be implementjed (at 
a mintmal cost) in their colleges. Participants were generally uhfamiliar 
with the TEPFU publications and available information concerning the state of 
the ar'!?. Mbst*would like to become part of a network and their names are in- 
eluded with this report. * . 

The pj^lmary focus of the discussion involved how participants copld get 
the "how to*s** as to evaluation design and interpreting the results. Answers, 
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* to thfe following (Questions and/or statements would be helpful to this group 
of participants , >^ - 

^ ther^e (should there be) some staOcTardized data from\fol low-up 
studies so'that one can compare^ a program' to a standard? " , 

'+ Can >eta" analysis be performed on data already gathered? (Is there 
enough data?) ■ ^ 

+ Are," or can there be^ both norm referenced, and criterion referenced 
data col lected on fol low-up studies? If so» what are some pro- 
: . cedures? ' ' ^ 

+ Are there standards or statements of intent by States as to beginning 
teacher performance criteria? Should states set the standards? Can 
we get this information? ' <^ 

+ How do we develop "reference points" for developing evaluation designs- 
and' comparing graduates from different programs? 

+ Should small col leges- have evaluation standards, designs* and expecta- 
tions different ^m large colleges? 

+ Oo current practices (such eis described or inferred in the paper pre-^^ 
sentatioris) provide "good" data for decision making? Have programs^ " 
actually changed'as a result of such decision making? Is it really' \ 

worth the effort? - 

i ' , ■ 

Should outside evaluators^e used to conduct fol low-up ^studies* rather 
than program personnel who» in some cases* are the program? 

+ Can NCATE -5t^ndar4s_^ madified for prograhis of different size and 
, emphasis? (This item vlas heavily discussed and there <M[s general* dis- 
satisfaction ylth rule 6 of the NCATE standards as small colleges ' 
would hav^ tt'ouble [T»6e)^ng all ^criterij^, gSome colleges have , already 
decided to pull out of the NCATE accreditation process because of 
these -requirements 0 „ ^ ^ . ' 

■ Ti 

+ How does'the "teaching" environment of the graduates influence fol- 
l6w-up studies? How car/W account for this variable in our evalua- 
t1ons» assuming there Is 4 difference? 

+ C^n long range (4-7 years) follow-up studies provide us with phjlx)- 
sophlcal/and general program data that can be useful for decisjoh 
making and program modification? (The intent of the question appeared 
to center on the thought that the undergraduate program should have a 
longer-lasting effect on their graduates than one or twp years* arid we 
need to measure this effects) * - * 
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The geheral tone qf the discussit/n and questions appeared to center on 
the need for participants to learn more of what is available, to share basic 
information, and to reinforce one another in the efforts to design sound fol- 
low-up studies and to use the d,ata intelligently. Participants expressed the 
need for some *'han4s*on" state or regional workshops . that would help them 
answer the questions and^ develop acceptable evaluation designs* 

The pVagmatic needs expressed ih this session included the need to im- 
press their administrators to provide additional funds for ^valuation pur- 
posesr. This included the notions of training faculty as well as providing 
resources to conduct the follow-up studies je.g travel , computer time, re- 
leased time, i.etc). Perhaps TEPFU, vis-a-vis the Texas R&D Center, can 
develop an "evaluation- program development kit" specif ical ly des igned for 
fol!ow-tip Uudies, and^lnclude cost estimates for various activities. .This 
along with suggestions as ^o how to finance such projects appear to be needs 
as well as suggestions for further activities. This writer suggests that 
perhaps a whole teacher training, program evaluation model (s) be developed 
which includes follow-up studies, \t would appear that the state of the art^ 
"is limited at this time, ■ ^ / > ^ 

V 
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^ ' so WflAT IS EVERYBODY TELLING US ABOUT ^ 

EVALUATING TEACHER EDUCATION? 

Joseph Vaugkan , ^ * 

* ■ National Ins.tttute, of Education 

U. S., Department of Education'* , 

Wby Do We Need It? ( , - - 

Studying the six papers and eight small group r^epor^s from this AACTE 
session reminds (ne of the story of the blind man be1hg placed" next to an 
elephant and/trying to figure^but exactly what it was. His interpretation^ 
depended upon where he touched Jthe beast and to which- of its very dJffere/it 
parts or dimensions he 'w$s expoVed/ So it appears to be with leacher educa- 
tion' evaluation^ whether one considers* it a , "beast" or a ''beauty." We all 
have our own interpretations of what it should do and how Mt should do \\. 
Some may look at teacher education evaluation and see U as principally de- 
pendent upon follow-up evaluations of\graduates while others will view it as 
more of an internal review process which includes teacher' education faculty 
and administrators and other advisors iff a sort, of introspective professional 
vigilance. Ask a dozen pecfple and. you'll probably get a Uozen different in- 
terpretations, each with thfeir pwn personal emphasis* .ReJ^ting that to the 
story, it could be that, we all may be at least partially "blind" to the total 
complexity and potential of 'evaluation .in teacher education and- thOs find 
ourselves "feeling our way*' and interpreting and emphasizing pieces rather 
than. the whole entity.. Hall suggests this in on6 place in .hls^paper and I 
would like to expand tfrat notion, exarnining the reasons for past and'^esent 
weaknesses In order to derive suggestions -of themes and processes for teacher 
evaluation* Twill base these suggestions, for th6 MSt part, on informatiqp 
from the critiques and recommendations raised in tfie group- of writings for 
this sessioot and on rny interpretations of the writings of others wjjo have 
examined this topic* To a lessjer extent, I shall be ,tresumptuous enctugh .to- 



... 

*Wh11e the author is* an ermployfe^ of , the National Institute of Education 
opinions expressed herein are so1e>y h^s own and do n&t necessarily represent 
the views of NIE or the U*S. Department of Educ^itljon. 



add fny personal thoyghts, identifying them as such. After all I'm fascinated 
wit(j elephants, too! 

Perhaps the. first reality we must al.l/face is that the need for evalua- 
tion o1* teacher education is not going to go away. ItMs not only, as Gubser 
so succinctly articulates, a -case of increased emph^s'^s and guidance being- 
put upon evaluation by NCATE, J8ne does not have to be clairvoyant predict 
where public disslatisfaction with education and teachers and the subsequent 
variety of state and locally initiated accountability rpeasures wi>l^ prol^ab]y 
lead us/ Just as local educatfon agencies and the teachers within them are 
being publicly ^taken to task-- Justifiably or unjustif iably--f«r perceived 
inadequacies, invest i gat ing some of the causes of thos^ perceived inacle- 
quacies must include close examination of teacher education quality and quan-' 

tity in process and content. The recent (April/May, 1981) series of Newsweek 

• — "~ — ^ 

articles on^ thf problems of our schools speak of teacher education as '|gen- 
erally, a hodgepodge of academic work, fieldwork and methodology whith can be 
helpful tricks of the trade^or irrelevant nonsense*" The articles refer to 3 
56% failure rabe among teaiher applicaotj on the National Teacher Examination 
in one .st,ate|^and in alfothpr,, a 47% /ai lure rate even^ though passing /was set 
at' about the 25th percentile* Weaver's ^work (1977) indicates ihat teacher 
education is attraoCthg ^ less academically able popula'tjon of students and 
much survey data exists to show us that teacher education inst4tuiion^ have, 
done little to combat this trend or to establish^ anything^ more than minimai^ 
standards /or admission artd retention of teacher education students (AACTE,* 
1981)* At tlie same'time, a Newsweek poU reveals that 90% of the respondents 
felt that- teachers should be require<f to pass a competency test of some, type , 
before they y?ehired< , We have a'wide gap which eysts between pub»Jfc and, 
mst of us would agree, logical professional expect^fions of at least minimal 
competenie^of teacher education gr*aduat'es ^nd the actual knowl^ge. base which 
thoSe.jjraiduat^s, possess. . \ - . ^ , 

While- the knowledge base whi^ih ii/assessed by mpst of the tests like NTE 
has not been^ sKown . ta b6 correlated with "?ffectiv|/teaching, logic mandVties 
^that we ^c.^t'the fact that' Tknowledgfe is , a;*jae^sary, although not suffi- 
cfent, condition (xf- teaching competence^ " CJearly,^ teacher education ' grad- 
uates as a jcoUective Bre not* rpeasurin^xip to wHat^we aa professionals must 



*demarid in the way (Jf quality. We do not know the>easons wKy this 1s ^o and 
that is exactly why we neied evaluation of, teacher education efforts and grad- 
uates. must understandX how we can improve the present condit.ions 'to*meet 
desirable and reasonable standards. Public dissatisfaction and professional 
obligation* will only i ncrease the importance of teacher evaluation in the 

i * 

future. 

What Knowledge Do ^e Already Have a^d What Else^Oo He Need to Know? 

■ ' The- Adams/Craig' p^er on a national survey of teacher educat.ion evalua- 
tion practices offers many insights^into what' U presently taking place. 
Although some type of evaluation is used in^ almost 9*o***^of 10 institutions* 
the nature and comprehensiveness of these varies widely. >Clearly, dominant 
practices include heavy reliance orf questionnaires filled out by teacher e<Ju- 
cetion^ students; first year graduates, and to a. somewhat lesser, extent*- 
supervi5;0rs. Much l^s use is being made of direct classroom observatipns or 
media-aided work- sample observations, standard^ized " tests ^is measures of 
kaow>edge or other factors* "and feedback 'loops that tie evaluation ba<?k to 
program, modification and improvement* J^oney supporting evaluationi is mini- 
mal, with 50% of the institutions spending less than $1,250 annually for this 
purpose. Most evaluation is focused on, the periods of* preservice and first- 
year teaching foUtfW-aip* Many institutions had major concerns with "develop- 
ing in^ticumentation/' "identifying instrumentation/' and "planning an evalua- 
tion system" in- addition to the feedback problem /nentioned "earlier.. Finally,- 
"Teaching Skills, Strategies and Techniques," "Knoy^le'^ge of Subject Matter"^ 
and "Relationships with Students" were areas of prime concern in the evalua- 
tion efforts. ' / , ^ ■ . 

This portrayal of the status quo' is very enlightening. It reveals 4 
rgXiance on superficial techniques tffat are mlfi^imal at beat» a dear'th of more 
comprehensive techniques (e,g, , direct classroom obSeryations)^ and very 
little financial commitment to evaluation* While^ teaching skills arje identi- 
fied as prime outcome (Concerns, very little u^e >S' made of evaluation me^- 
-sures that can yield valid and reliable data on these outcomes. For example, 
it Is e)itre1iiely difficult to imagine^ a questionnaire that'^would yield accu- 
rate and useful information on teaching skills, especially when the student 
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Or graduate her/himself is often the person being asked, Subser also poi^jts 
out that return rates of 15SUon such questionnaires are riot uncommon. While 
we know from prior research on instruction that direct classroom observations 
and/or media-aided work samples are imperative, to Understanding teaching and 
teacher effectiveness, these measures are glaringly absent from most teacher 
education student/graduate ev*aluatioos. One is led to believe, as was raised' 
in one smalT group report, that cost considerations are the primary reasons 
for this absence, but more on that later. ^ . 

Subser makes an extremeTy significant point about the differertces b^- 
tween and the-need for both graduate and program evaluations and*how the Ut- 
ter seem(^^ very fhuch overlooked. He reminds us of the need to assess course 
sequences and content, textbook nature, and use,-, admission and retention 
standards, and otji^ program features as determinants of student acquisition 
of skills^ kna<Jledge, attitudes and behaviors while in preserVice ,* He also 
discusses thel importance of the roles sind .nature^ of Involvement of teacher 
education facurty and administrators in planning, implementing, and using the 
results of evaluation activities. At the same time he warns us of the dan- 
gers of using graduate "success" or "competence'' as surrogates for program 
"success" since we know very little about how much *of the^ preservice effect 
"washes out" as a result of the unique influences of different school dis- 
tricts and schools in which ;gradu(Stes take positions. As Hall tellJ us, we 
must aTso be aware that the Jonger th6.. graduate ■ is fn a regular teaching 
position, the more likely it wilKbe that preservice eff^ts are less potent. 
Certainly^ graduate^ effectiveness has to be related to program effectivjenes^ 
but they are discrete entititeSt which must also be understood separately be- 
fore interrelationships can. begin to be drawn with any "degree of confidence. 
To date, there has been very little emphasis on thesfe important distinct 
^tions, ■ . - ^ 

Certainly, the concept of a teacher education continuum of which .Hall 
speaks is as essential element In ultimately understanding the influence of 
preservice and inservice experiences and effectiveness* While each stage has 
its separate influences there is thje ultimate need to determine how they are 
interrelated. This can,b^ paired with recent emphasis on. the need to examine 
this "interrelatedness ^during the 1-3 year induction pgriod of beginning 
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teaching. Practitioners and researcher^, ^Jike'have identified induction as a 
period of high trauma and significant formative inf.luence in how teachers 
will teach^ throughout their career^^'or* in many cases* whether op,,jiot they 
even want to st5iy in teaching. While there os no l^fck of qtfantjty of data 
b.eing Tol lected on ^this period* especi al^^ {he tirst ^of teacfiing-, the 
quality and comprehensiveness of the dat^jre suspect r ^; ; . " ' 

- Given these- overview papers » Ihe^Pree papers , by AyerS»/Tfearn an<J 
Craig/Adams offer useful. » institution-specific cases, which relate directl^;^ to 
the broader points of. the other* authors. The two institutions^descrijj^ed are 
of particular interest because they are» io many res pec ts^^j^^-Ur^^^ 
those trying to systematical ly address teacher education^^evaluation , " But 
they are also, I believe^ in, some respects plagued by the unsolved problems 
that' are faced by the vast majority oV teacher education institutions whicjj. 
are looking for help in designing and implementing evaluations, ^ " - 

Ayers and Hearn discuss efforts to develop ■ a comprehensive xnotf^T^ of 
graduate, fol low-up. evaluations and have incorporated many deflrable aspects 
mentioned in the three .overview papers, The ten princip^^ mentioned by 

g 

Ayers, reflect sound thinking about planning* implementation; use and revision 
issues in such" evaluations, Key positive fe^ures include: , 1} the need for 
both Clarification of specific and general program objfictives^ and relating 
Evaluation to these; 2) a continuous dnd longitudinal design; 3) Qgncerfr with 
adva^^ments in evaluation methodology; 4) early and close collaboration .with 
schools; 5) use of a variety of data collection techniques and sour/es and 
wllection.of both qualitative and quantitative *data; and 6) .a feedback pro- 
cedure to make ev^aluation results knawri.td facou]ty and administrators, ^ Heara 
adds the element of sufficient financial resotJhtfe' allocation to the activi- 
ties and -^presents an -excellent stx-poiht analysis of l<ey considerations .in- 
cluding use of field'based faculty as datfii ^61 lectors/ concern with the en- 
tire- preservice/induction/inservice ^:ontinuilm» data sampling and protessing 
issues » and multi-institutional consortia to' pQol resources and^ reduce 
C05tS* * . ■ 

While these are indeed positive and commendable features, they are in-^ 
Ayers' words, "sti.ll incomplete Snd inadequate." (To some" extent , one always' 
faces the pos'Si.bility that^papers may simply not .address some issues duetto 
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lengtKrestrlctions rather tfiaD by conscious choice. I apologize In adV5(hce 
for citing, concerns about ^hese programs \rfiicti may have been adequately a(f- 
dress§[d hat^ the design of the papers been more amenably to telli;ig the insti- 
tutipnfi; fuTl stories*) ,/or example,, while the destrability of specifying 
Objectives is unquestionable it is equal ly *clear that /much must also be fcnown 
about the fextent to which ,the teacher education practices maetched the stated 
objectives*/ To move, from objectives to evaluation .without cons.iderat ion of" 
impTerugntation produces, a tenuous link at best,.," On the evaluai_ii)n techniques 
issues, %ers appears to confuse the proges,s made in techniques for research 
on e^^tective teaching with techniques for effective evaluation: The former^ 
is very important as a determinant ■ of the content! of teacher ledjucation it- 
self, but it5; linkage with evaluation processes, is never specified. It is 
also import ant ''to consider, as, far as school collaboration is concerned; how 
we can go beyond the .I^permission" and '^understanding" levels to. involving 
schools in t^ desigrt, implementation and use of evaluation mechanisms which 
'will be of a? much benefit to^school practitioners as to teacher educators* 
As far as program modification is concerned, the papers fail to^ address how 
the feedbaclJ Joop is created and how the nature and extent of inodifications 
IS monitored Mt it not sufficient to communicate results to faculty and ad- 
minist?rators* Vhere must be. fpllow-up to see that something constructive 
^happens" through individual or group motivation as a result of the feedback* 
While Hearn'sfdelineatjon of'the importance and nature of evaluation costs is 
Helpful , his \sugge^Jion that costs could^ be signi ficantly reduced through 
el4j^inat1^ of observations is not appropriately qualified by, the imm&nse 
dangers of so doing. I v^puld suggest t^at this element is at the heart of 
any potentTal contri-bution that could ije made, and reliance on more super- 
ficial data^ col le(^ion is indefensible at this time* We simply do not have, 
less expensi.ve. alternatives with which we* can correlate' teacher skills and 
performance*'" Finally, I would like- to contest that principle number 9 in 
Ayers*, paper may be the reverse of what should be the Case. Specifically, is 
it not the case that progKam decision-making mu&t provide for a workable sys- 
tem of ^evaluation and evaluation use rather than the <ither way around, as is 
stated? . Evaluation is a part^of program planning and implementation, not the 
reverse* It ma^ b^ a subtle distinction but one of great importance in con- 



sideration of how evaluation can, be fully integrate* with pr^ogram design, 
develbpment, operation df^tf modification ; , * - ^ 

; ' -The Craig/Adams pSper shows many simDarities to the Hearn and Ayers* 
descriptions, accurately reflecting the^ fact that there ha? beert much collab- 
oration and mutuality in the separate institution "^s development of ^ evaluation 
efforts. In additiofi; Craig'^Adams address the socio-political context within 
yvjhich tV^hgi" education evaluettions^exist- They also discuss the issues of 
, uie of such evalua'tions and th£ objective/subjective interpretation of re- 
sults. A key concern is .the hieVaVchical structuie and conditions for making 
decisions about-the nature and use of evaluations-- The hierarchy emphasizes 
interactlocL and collaboration an^Dng teacher education faculty and administra- 
^ tors^.and advisory functions .fran school personnel^ The ideal Qfocess for 
plannirtg» implementing ^nd using^ev^lifations is seen as being based on ''in- 
volvement^^ and "evoWement. " Key final ^ steps are check i.ng* the extent to 
which evaluations are used to make program modifications and (tjpterminin^,^on 
a continuing basis, >ow the evaluation process e^n be improved- ^ 

Certainly the .principJes espoused in the Craig/Adams paper are laudable 
and much. QT the structure which I assume they have set up seems potentially 
appropriate to acccHnplishihg their objectives/ There £)<? appear t^K^sT^ how- 
ever* alternative sequences of activities and combination$\pf>lJeasiofi-makers 
which might be considered in attempting to strengthen the. eval uation program 
arTd, in particular, its impact^ Initially, the assumption that one must 
start' with the chief administrative off tcer^{e-g., dean) to perform^ such 
functions as determining the *'ideal program^ need optiOftS-^-^ implica- 

tions could be questioned. Why not dd this with a col laborat/ive group, 
inc Hiding the chifef -administrative officer, ri^ght fKpm the begi nning? In 
lhat fashion, a system of checks and"bala«ces would-be in place^ **ov(nersli^p" 
wouljd ' probably be enhanced, diverse perspectives would be ensured ' and, none 

^^of the benefits' of the Chief administrator's sole determinations would, be 
lost- What this really would amount to would be a reduction of three layers 
of ^ decision-making inter ^one y/ith the chief* adm'inistrative officer, decision- 
\maTcing learn and planning/evalufl<Eion team memberships being represented on the 

H composite teahi. The. way the Craig/Adams hierarchy appears, there are*- two 
layers of deQision-mak'ing before the "primary tnitiation and operational 
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force" (planning/evaluation team) brought into the picture* To separate 
establishment of ''parameter^" from "initiation" witf) two layers of 'decj$iQfi 
bears the possibility of being perceived^as a camouflaged "top-down'^ policy 
mandate. About th$ only reason I can perceive' for oot involving the ro1e 
groups as collaborators from the beginning is.th^ difficulty of lar^e^ num- 
bers» but it would appear that hard .negotiation of a representative groiip*^ 
would bi small payment for ensuring removal ^of the ,potery;i^l ^"top-dowri" stig- 
ma . ^ ^ ' ^ . . , ' ^ 

Alonj witt) tfte above -concern there are only two other reactions *which 
jump to mind from ffift Craig/Adams paper, >Fir'st» the delicate issue of the 
role of school pracTittoners has^ be^ addressed in the paper .thrpUgV having 
them in an "advisory" capacity, "Siven'^the realities of pow$t; structures ih 
higher education* it is perhaps understandable that scl^ool persofihel/ Could 
"not be given parity status in. a formal sense. "^8ut, classifying them as "ad- 
visory" could portend les^ than maxjmal involvement and perceptions that this 
input is more or less an "add-pn,"^ Any measures Ijhat ^ould be utilized to 
ensure parit^y status* with or withput formal position, should be encom^aged. 
As most of us would agree* we .are grobably nx)re likely to feel committed and 
contribute to an effort when we feel that our voice is of equa.1 volume ^and 
impacts as those with whom w6 are' working, ''One other point deserves, mention* 
with regard to school personnel parity, is those schools which are the 
"buyers/' in a "buyers market," Does it not behbove teacher education person- 
nel» and seem very reasonable* if ortly^foV ^suryival* to pay full attention to 
the input of school personnel relative \o 1); existing and changing contexts 
within whicVi teacher education graduates v/jll operate^ and 2) the expecta- 
tions and processes used by the "buyer" in determining which teacher educa- 
tioq. graduates are' desirable and pot'entiaT job-holders? ' 

A second cor^cern is with acceptance of the status quo "organizational 
frafliework" as the det^minant of options, * Realizing that one must acknow- 
ledge what exists and its limitations; that does nofpreclude working toward 
more desirable structures or frameworks, -There is' a *'self-ful 1 ing , pjrophecy" 
dimension to assuming you can't chartge simply because present structures 
won't allow it» just as the[re is' in assuming mo new sources of funds are 
1 ike ly for evaluation* One major ^difficulty /identified and discussed by 



many* {Smith,. 1980; Oenemark, 1979, etc.) is the nature of the present struc- 
ture of teacher education* Without dc^stic >changes in area's sach as incen- 
tive structures, funding formulae from tt\e state Ifivel, the col lege ujf. educa- 
tion's control of its own governance, etc., we probably have less trope for* 
exploration -of major alternatives to present" practice, including , the area of 
evaluation and its impact* It a case is to be made for such changes (or at 
least the structural flexi])ility to consider them), we cannot afford .a stance^ 
of passive acceptance of what exists. There is much worth keeping and much^ 
that needs examination, and^ change/ Our obligation would appear to be to">work 
toward^ the structures within which such considerations and action are pos- 
sible. 

1 5nust 5ay, climbing down off my soap box, that the irttent and substanC? 
of what Xraig/Adams, J^yers and Hearn recommend are^ very ^much worth noting* 
My questions have to do with how everything can be most effectively and ef- 
ficiently accomplished* Both institutions should be loudly applauded for 
their efforts to, date and both serve as valuable guides to, other institutions 
seeking possible answers to very difficult questions* 'My comments on their 
papers, as well as those of Subser* Hall^and 'the survey work of Craig and 
Adams* are intended to be as constructive as possible and with full recogni-^ 
tion of and respect for the considerable work and thoughts evident in each 
paper, \ • * 

What Are Some* "Best Guesses" At This Time About Where We're^ Headed in 
Teacher Etiucation Evaluation Efforts? 

I would like to talk a bit about what we are likely t<jf see happening in 

the near and distant futur^ and how that might compare wi^h what some might 

consider "ideal" conditions. In order to do thtSf it apj^ears to me as well 

as to many of the writers,* that we must consider not on)y follow-up studies 

but the whole spectrum of teacher education program and product assessments 



from the beginning of undergraduate work through some ^ecif ied^period after 
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graduates are teaching in the schools* Without thi&^ fonceptual^iza 
concern for the total effort over a continujum of time, the basic pol 
Hall, Gubser and several others will go unaddressed* j 
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' \ . -In^" considering- the future, I^wilUtry-to incorporate the *respopses of . 

'.th§.,eight small group- discussions along with myt own thoughts, I would" be 
verjj refni^ss if, I did not comment on. how inipres?sivQ tl^ brief^small group re- 
pdrts were, giuen the stiort Hime they liad^o work and 'the complexity of the 
issueSrf^fJhi le ^the reports Indicate" that the top^ic it*i%l-f,.f is one of great 
curri^ency» interest dnd controversy, they al^o indicate 1:he considerable po-" 

, tentiaVof this type of interaction in prcnjucing worthwhile thinking and sug- 

^■ge^^ions. The ^cfiver^ity and excellence of thought is hearjteninjg, 

^ Sej^qral 'ofsjihe cojfimentY glre^dy made have obvious implicatiorfs for the 
\* - : ■ ' / ' ■ . ^ . * ' 

fyt^e, *lt Is not tiecessary :to reiterate those in-depth, .onl>e to sumrtarize 

that the core of pi^esent and future^teacher" education evaluation .efferts must 

be formed by. more comprehensive, detailed and objective data collejctlon; col* 

laboratipa on design, imp-Jementation ^na use; concTern with b^th' ^^rogram and 

products;, ajcfequate financial support; and a longitudinal perspective. It is 

also 'clear that this is frequently not the ca^e in existing* efforts. The 

srr^]] gVoup Ve'ports give "i/s a. brOader perspective on the reasons why, as well 

as what is needed to i)egin moving^ toward improvement of the qualit-jj; an^^quart- 

t.ity of evaluations- I shall refer to the reports by the name of {he discus- 

sion leader in the group, . * 

, . Virtually all of the group reports substantiated many of the basic weak- , 
ne*iSes and strengths brought out by presenters./ But «ach group took its own 
dp^jroach in spying what that m^ant to them and how they Uw themselves ad- 
dressing tt>e problem^ and capitalizing upon existing "5uCces$V and knowledge. 
Perhaps the central concern , of the groups was best stated ^y the Dravland 
coriVneht that what was needed, was "a well articulated framework within which 
we_j:an define o*Jr 'tasks and- our areas of ignorance^^' 

The White report emph^asized the n6ed for initially considering the match 
6r^ mismatch between professional community needs" and "prograirf^ outcomes" 
and, vrilhin that, the critical pbin^t of^ identifying" instructional and program 
Objectives, One difficulty with that, as the Froyen group discusses, is that 
the ^faculty who comprise . much of th^t community are highly resistant to 
change, often in the name of academic ftfeectofli and autonomy, Froyen contends 
that little attention, is paid to consensus-building in developing objectives 
and even less attention to standards, of compliance with those objectives. So 



we ft^ve a built-in conflict in establisMfjfl so^nd fourxfation fo*^'*deve loping 
reasonable and appropriate evaluation <\ueJ^tyor\s and processes. In ^the-,pa5"t 
this has often resiilted nn no action., ^f^ortunately, (or unfortunately^ de- 
pending on your perspective}, there are 'o'ow accountability forces in effect 
that may. necessitate action. Froy^n's excellent list of -reason? commonly 
given about why we can' t do Quality Evaluations are likely to be overwheln\ed 
by the state legislative and statls' dep^irtment of education nrandates» prcrfes- 
si-orKaj) association activities and accrediting agency demands for **^proof ^hat ^ 
, teacher education is doing its job satisfactorily.- In short, mofe stringent 
measures of ^'success" will be mandated and» put most -simply, it is logical tp 
assume that university faculty and the r^st of the professional community 
would' rather get involved and ha\^e some^say in tfieir own destiny as opposed 
to having their fate totally determined by other external forces. This is. 
perhaps not the most posit.ive or altruistic reason for becom.ing^ involved » but 
it is likely lo be. a,reality. Once we move beyond specification and' clarifi- 
cartion of objectives^ find ourselves in a position of having to k/iow what 
pr^sentl^ exists in teacher education evaluation practices,^ A common infor- 
mation base is a necessity if we ara to avoid Veinvent'ing the wheel/* as the 
deVoss group di§cusses» and capitalize upon past successes and failures. The 
kinds of (Questions that ^must be asked and for which information^ must be 
gathered are typtfied irr the Kilgore report. What ar.e present stand^ard^s far 
graduates which are being mandated by_ various^ states to which Qur students^ 
might go for a job? How does NCATE Vit in with all this? Have some teachi^r 
education programs actually changed as a result Qf evaluations iand» if so». 
how? How do we look at the impact of preservice given the uncertain effects 
of each unique school setting? These ir\6 other questions posed by Kilgore 
are somewhat scattered but that, is exactly the real world for frost teacher 
education institutions . Institutions have been too isolated and need to 
become^ aware of the state-of-the-art of evaluation arid the context witt^in 
whicfr that evaluation must exfst. As White states^, we need descriptions o%, 
^'evaluation programs currently in place or binder development with a discus- 
sion of conceptual features^ operational preconditions^ . implementation prob- , 
lems and benefits^ cost effectiveness and» most importantly^ what difference 
has been made as a result of the prog/am/' ? " ' 
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.In terms of/ the actual design and conduct of evaluations, marvy good sug- 
gestions were/made by the groups. For example, the deVoss paper contains 
several creative ideas including the use of volunteer teachers sitting on- 
camptiS wtxh faculty groups to discuss evaluation issues, documentation begin- 
ning frarly in undergraduate years, "critical incident" logs being kept by 
teac>ier education students, cooperating teachers and supervisors, and intense 
data collection with smaller random samplings of students/graduates, NCATE 
itself, deVoss suggests, has been a somewhat unintentionally negative influ- 
^encfe'^due to its demands for wha^t jnany institutions see as a limiting view of 
evaluation. At^ the same time, it has the potential to be a very positive 'in- 
f luence if it could play a more active role in disseminating information^ 
■^bout s'^uccessful approaches-^ aod encourage experimentation and diversification 
in e\/aluating programs and products. Certainly, the Gubser, presentation in- 
.dicates very harmonious thinking along those lines and, perhaps portends just 
such activity. Again, a basic issue to be addressed is- exactly just what 
constitutes *'prdof** that teacher education is doing its Job and to understand 
teacher' education's limitations as well as its potential. The Savage group 
^ (sounds menacing, doesn't it?) reminds us that the systems model, used in .in- 
dustry guarantees the '^product" only, as long as it is used^ as intended. The 
potential mismatch Ibetween "IdeaV' "preparation and the idiosyncratic reali- 
ties of individual ichools mandates that ^institutions of teacher education 
closely examine their objectives and programs within that real world context. 
The "proof*^ indicators must emanate from a realistic assessment of the rele- 
vance and impact of prfeserviCe on what a teacher does day-to-day in a class- 
room, school and convnunity environment. The idea of -more active Stete in- 
volvement in teacher evaluation, as in the Seolrgia example cited in the, 
Savage paper, must be inve^ti^ted and understood in relationship to the 
evaluative rol^ of the higher education institutions ancT the schools* 

Finally, in terms of the u$e of evaluation results. Hungerm^an raises a. 
very important point about the need to formalize processes for communication, 
d.iscussion and modification of' program if we^ are to reaVy ' bring about 
changes. To just disseminate the results to faculty and administf'ators is 
not enough. It is extremely important, the r*eport goes on, to have a visible 
and acknowledged (and,. I would add, respected) leader for the evaluation ef- 
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fort to maximize impact. It strikes me" that one alternative to having tjo 
choose between an evaluation or program "type" for this role is to considi^r 
dual leadership where both roles would be represented* Through this ine^s, 
credibility is likely to be Increased along wUh the overall relevance;'^ and 
validity of the effort. Garland reminds us, as did others, that whatever re- 
sults are uspd, they must reflect evaluation of the total tea^cher edMcation 
leffort from admissions to program replacement and beyond. Only in that way 
can we consider evaluation truly. systematic, ^ 

In summary, we have our work cut out for ust We've identified a great 
number of seriojjs weak^nesses but we've also identified many useful approaches 
and principles worthy of consideration. The time has come to move beyond 
stating the problVns and to look at how we can build upon the conceptual and 
structural necessities and beginnings discussed with much insight in these 
papers. We need not be too cynical or pe-ssimistic. Thft session reveals a 
valuable pool of knowledge and judgment about what is needed and how we can 
begin t& move toward" those enJs. It may be a /difficult task ^we are 
ufidertaking but it* need not be averwhelming* Awarelics-^-has been created and 
we now must move toward action . This work stands as afn excel lent 
beginning. ' * . 
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